Oct 30 19:01:17.813423 ip-10-0-140-235 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 19:01:17.813436 ip-10-0-140-235 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 19:01:17.813446 ip-10-0-140-235 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 19:01:17.813790 ip-10-0-140-235 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 19:01:27.554995 ip-10-0-140-235 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 19:01:27.555009 ip-10-0-140-235 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 4f9335b3918d4333a3f7cd383854f7ad -- Oct 30 19:03:25.282350 ip-10-0-140-235 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:03:25.736900 ip-10-0-140-235 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:03:25.736900 ip-10-0-140-235 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:03:25.736900 ip-10-0-140-235 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:03:25.736900 ip-10-0-140-235 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:03:25.742631 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742534 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:03:25.742631 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742631 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:03:25.742631 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742635 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:03:25.742631 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742638 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742641 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742644 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742647 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742651 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742654 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742658 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742661 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742664 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742666 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742669 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742672 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742674 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742677 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742684 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742687 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742690 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742693 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742696 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742698 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:03:25.742759 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742701 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742703 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742706 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742709 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742712 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742714 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742717 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742719 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742722 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742725 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742727 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742730 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742733 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742736 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742741 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742746 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742751 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742754 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742757 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:03:25.743255 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742760 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742762 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742766 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742771 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742775 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742779 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742781 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742784 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742787 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742790 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742792 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742795 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742797 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742800 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742802 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742805 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742809 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742812 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742814 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:03:25.743709 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742817 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742819 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742822 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742825 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742827 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742830 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742833 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742835 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742839 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742842 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742844 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742848 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742851 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742854 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742856 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742859 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742862 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742865 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742868 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742870 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:03:25.744165 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742873 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742876 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742878 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742881 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742884 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742886 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742889 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742891 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742894 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742896 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742899 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742901 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742904 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742906 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742909 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742912 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742914 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742917 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742920 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:03:25.744638 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742923 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742928 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742930 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742933 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742935 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.742938 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743743 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743750 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743753 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743755 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743759 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743762 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743764 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743767 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743770 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743773 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743775 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743778 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743780 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:03:25.745161 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743783 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743786 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743789 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743792 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743795 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743798 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743800 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743803 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743806 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743809 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743812 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743814 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743818 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743821 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743823 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743826 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743829 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743831 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743835 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:03:25.745632 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743840 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743843 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743846 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743849 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743851 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743854 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743856 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743859 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743862 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743864 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743867 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743870 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743872 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743875 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743877 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743881 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743883 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743886 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743889 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743892 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:03:25.746114 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743895 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743898 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743901 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743903 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743906 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743909 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743911 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743914 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743916 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743919 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743922 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743924 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743927 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743930 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743933 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743936 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743939 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743942 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743945 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743947 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:03:25.746595 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743950 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743952 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743955 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743958 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743961 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743963 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743967 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743972 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743975 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743978 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743980 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743983 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743986 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743989 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743991 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.743998 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744001 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744018 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744021 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:03:25.747085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744023 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744026 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744028 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744032 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744034 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744037 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744040 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744043 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744046 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744049 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744051 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744054 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744056 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744059 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744061 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744582 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744589 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744592 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744596 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744599 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:03:25.747554 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744602 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744605 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744608 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744611 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744613 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744616 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744619 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744621 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744624 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744627 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744630 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744632 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744635 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744637 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744640 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744643 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744646 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744649 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744651 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744655 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:03:25.748055 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744658 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744661 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744664 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744666 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744669 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744672 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744674 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744677 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744679 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744683 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744687 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744689 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744693 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744696 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744698 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744701 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744703 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744706 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744709 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:03:25.748526 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744712 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744714 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744717 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744720 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744722 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744727 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744731 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744735 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744738 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744740 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744743 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744746 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744749 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744752 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744755 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744757 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744760 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744763 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744765 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:03:25.748997 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744768 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744770 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744773 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744775 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744778 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744781 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744783 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744786 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744789 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744792 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744794 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744797 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744800 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744802 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744805 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744807 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744810 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744813 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744817 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:03:25.749460 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744820 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744822 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744825 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744827 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744830 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744832 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744835 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744837 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744841 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744843 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744846 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744849 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744852 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744854 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744857 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744859 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744862 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744868 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744871 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744873 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:03:25.749940 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744876 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744879 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744881 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.744885 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747281 2574 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747292 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747300 2574 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747305 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747310 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747313 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747318 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747324 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747327 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747330 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747334 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747337 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747340 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747343 2574 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747347 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747350 2574 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747353 2574 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747357 2574 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747363 2574 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:03:25.750423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747366 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747369 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747372 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747376 2574 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747380 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747384 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747388 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747392 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747395 2574 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747399 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747402 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747405 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747408 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747413 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747416 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747419 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747422 2574 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747425 2574 flags.go:64] FLAG: --enable-server="true" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747428 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747432 2574 flags.go:64] FLAG: --event-burst="100" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747436 2574 flags.go:64] FLAG: --event-qps="50" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747440 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747443 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747446 2574 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:03:25.751002 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747450 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747453 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747457 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747460 2574 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747464 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747467 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747470 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747473 2574 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747476 2574 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747480 2574 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747483 2574 flags.go:64] FLAG: --feature-gates="" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747488 2574 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747491 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747495 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747498 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747501 2574 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747504 2574 flags.go:64] FLAG: --help="false" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747508 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-140-235.ec2.internal" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747511 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747514 2574 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747517 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747520 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747524 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747527 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:03:25.751734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747530 2574 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747533 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747536 2574 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747539 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747542 2574 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747545 2574 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747548 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747551 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747554 2574 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747557 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747560 2574 flags.go:64] FLAG: --lock-file="" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747563 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747567 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747570 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747576 2574 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747579 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747582 2574 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747586 2574 flags.go:64] FLAG: --logging-format="text" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747588 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747592 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747595 2574 flags.go:64] FLAG: --manifest-url="" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747598 2574 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747603 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747606 2574 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747610 2574 flags.go:64] FLAG: --max-pods="110" Oct 30 19:03:25.752328 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747613 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747616 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747619 2574 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747622 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747625 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747629 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747631 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747639 2574 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747642 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747645 2574 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747649 2574 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747652 2574 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747654 2574 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747657 2574 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747660 2574 flags.go:64] FLAG: --port="10250" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747664 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747666 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02cb9b13e1dc18340" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747678 2574 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747682 2574 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747685 2574 flags.go:64] FLAG: --register-node="true" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747689 2574 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747693 2574 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747696 2574 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747699 2574 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747702 2574 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747706 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:03:25.752919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747712 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747716 2574 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747719 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747722 2574 flags.go:64] FLAG: --runonce="false" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747725 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747728 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747731 2574 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747734 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747737 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747740 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747744 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747747 2574 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747750 2574 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747753 2574 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747756 2574 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747759 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747763 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747766 2574 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747769 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747774 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747777 2574 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747780 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747785 2574 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747790 2574 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747793 2574 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:03:25.753588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747796 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747799 2574 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747802 2574 flags.go:64] FLAG: --v="2" Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747807 2574 flags.go:64] FLAG: --version="false" Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747811 2574 flags.go:64] FLAG: --vmodule="" Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747815 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.747819 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.747959 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.747963 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.747967 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.747970 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.747972 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.747975 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.747978 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.747980 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.747983 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.747986 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.747989 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.747991 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.747994 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:03:25.754230 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.747997 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.747999 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748012 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748016 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748019 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748022 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748024 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748028 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748032 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748035 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748039 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748042 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748045 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748048 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748050 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748053 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748056 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748059 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748062 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:03:25.754719 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748067 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748070 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748073 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748075 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748078 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748081 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748083 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748086 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748089 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748091 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748094 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748097 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748099 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748102 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748104 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748107 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748110 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748113 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748115 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748118 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:03:25.755191 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748121 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748124 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748126 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748130 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748135 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748138 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748142 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748145 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748148 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748151 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748154 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748156 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748160 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748164 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748166 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748169 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748172 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748175 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748178 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:03:25.755666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748180 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748183 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748186 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748189 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748191 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748194 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748197 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748200 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748202 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748205 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748207 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748210 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748213 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748216 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748218 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748221 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748225 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748228 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748230 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748233 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:03:25.756142 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748236 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:03:25.756617 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748238 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:03:25.756617 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748241 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:03:25.756617 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748244 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:03:25.756617 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748247 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:03:25.756617 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748250 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:03:25.756617 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748253 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:03:25.756617 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748256 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:03:25.756617 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748259 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:03:25.756617 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748261 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:03:25.756617 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748264 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:03:25.756617 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748267 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:03:25.756617 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748270 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:03:25.756617 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748273 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:03:25.756617 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748275 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:03:25.756617 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.748303 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:03:25.756617 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748339 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748344 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748347 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748350 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748352 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748355 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748358 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748361 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748364 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748366 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748369 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748372 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748375 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748379 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748382 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748384 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748387 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748390 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748393 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748395 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:03:25.756991 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748398 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748403 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748408 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748411 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748414 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748417 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748420 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748422 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748425 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748428 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748430 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748433 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748436 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748439 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748441 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748444 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748447 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748449 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748452 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748455 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:03:25.757473 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748458 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748461 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748464 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748466 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748469 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748473 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748476 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748479 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748482 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748485 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748487 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748490 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748492 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748496 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748499 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748501 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748504 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748507 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748510 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748512 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:03:25.757946 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748515 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748518 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748520 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748523 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748526 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748528 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748531 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748534 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748537 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748540 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748543 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748545 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748548 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748550 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748553 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748556 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748558 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748562 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748564 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:03:25.758445 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748567 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748570 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748573 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748575 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748578 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748581 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748584 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748587 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748590 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748593 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748595 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748599 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748603 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748607 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748610 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748612 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748615 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748618 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748620 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:03:25.758918 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748623 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748625 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748628 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748630 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748633 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748635 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748638 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.748640 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.755850 2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.755866 2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.756546 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.756553 2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756653 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756658 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756662 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756665 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756668 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756671 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756674 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756677 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:03:25.759413 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756680 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756683 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756685 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756688 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756700 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756703 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756706 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756708 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756711 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756714 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756717 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756719 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756722 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756725 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756727 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756731 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756733 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756736 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756739 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:03:25.759890 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756741 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756744 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756751 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756754 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756757 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756760 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756762 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756766 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756768 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756771 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756774 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756776 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756779 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756782 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756785 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756787 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756790 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756794 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756804 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:03:25.760391 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756808 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756811 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756814 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756816 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756819 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756822 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756824 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756827 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756830 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756832 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756835 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756838 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756840 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756843 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756845 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756848 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756850 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756853 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756856 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:03:25.760846 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756858 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756861 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756864 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756866 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756869 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756871 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756874 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756876 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756879 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756882 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756884 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756886 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756889 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756899 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756902 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756904 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756907 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756910 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756912 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756915 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:03:25.761312 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756918 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756920 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756923 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756925 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756928 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756930 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756933 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756935 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756938 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756941 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756945 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756948 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756951 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756954 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756957 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756959 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756962 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756965 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756967 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:03:25.761797 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756970 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:03:25.762290 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.756972 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:03:25.762290 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.757001 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:03:25.762290 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757236 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:03:25.762290 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757244 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:03:25.762290 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757248 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:03:25.762290 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757252 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:03:25.762290 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757260 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:03:25.762290 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757264 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:03:25.762290 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757266 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:03:25.762290 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757269 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:03:25.762290 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757272 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:03:25.762290 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757275 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:03:25.762290 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757277 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:03:25.762290 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757280 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:03:25.762290 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757282 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:03:25.762290 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757285 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757288 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757290 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757293 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757295 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757298 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757300 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757303 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757306 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757308 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757311 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757313 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757316 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757318 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757321 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757323 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757326 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757329 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757331 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757334 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:03:25.762666 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757336 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757339 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757342 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757345 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757354 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757357 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757360 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757362 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757365 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757367 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757371 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757375 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757378 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757381 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757383 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757386 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757388 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757391 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757393 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:03:25.763146 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757396 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757398 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757401 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757404 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757406 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757409 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757411 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757414 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757416 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757419 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757421 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757424 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757426 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757429 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757431 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757434 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757436 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757441 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757449 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757452 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:03:25.763588 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757455 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757457 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757460 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757463 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757465 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757468 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757471 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757474 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757477 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757479 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757482 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757485 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757487 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757490 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757493 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757496 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757498 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757501 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757503 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757506 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757508 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:03:25.764116 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757511 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:03:25.764610 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757513 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:03:25.764610 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757516 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:03:25.764610 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757519 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:03:25.764610 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757522 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:03:25.764610 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757524 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:03:25.764610 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757527 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:03:25.764610 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757529 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:03:25.764610 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757533 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:03:25.764610 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757536 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:03:25.764610 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757544 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:03:25.764610 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757547 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:03:25.764610 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757549 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:03:25.764610 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.757574 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:03:25.764610 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757620 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:03:25.764610 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757625 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757627 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757630 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757633 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757635 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757638 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757643 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757645 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757648 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757651 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757653 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757656 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757658 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757661 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757664 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757666 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757669 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757671 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757674 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757676 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:03:25.764959 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757679 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757681 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757684 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757687 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757690 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757692 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757698 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757700 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757709 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757711 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757714 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757717 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757720 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757722 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757724 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757727 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757730 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757733 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757739 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:03:25.765782 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757742 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757744 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757747 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757749 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757752 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757754 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757757 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757759 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757762 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757764 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757767 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757770 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757773 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757775 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757777 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757780 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757782 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757785 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757788 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:03:25.766518 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757792 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757795 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757797 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757807 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757810 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757812 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757815 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757817 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757820 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757822 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757825 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757828 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757831 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757834 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757836 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757841 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757843 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757846 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757848 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757851 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757854 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:03:25.767054 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757856 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757859 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757861 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757864 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757867 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757869 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757872 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757874 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757877 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757879 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757882 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757885 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757888 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757891 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757893 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757902 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757905 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757907 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757910 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:03:25.767612 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757913 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:03:25.768239 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757915 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:03:25.768239 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757918 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:03:25.768239 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757923 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:03:25.768239 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757930 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:03:25.768239 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757933 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:03:25.768239 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:25.757935 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:03:25.768239 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.757991 2574 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:03:25.768239 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.760186 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 19:03:25.768239 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.761253 2574 server.go:1014] "Starting client certificate rotation" Oct 30 19:03:25.768239 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.761381 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:03:25.768239 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.762288 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:03:25.768239 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.763700 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:03:25.768239 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.767423 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:03:25.778775 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.778755 2574 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:03:25.782499 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.782482 2574 log.go:25] "Validated CRI v1 image API" Oct 30 19:03:25.783310 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.783282 2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:03:25.786489 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.786465 2574 fs.go:135] Filesystem UUIDs: map[4ea0243f-3872-4281-9afc-62a44536e3b6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a7a49493-b336-4bb8-8627-6a373f7aac9e:/dev/nvme0n1p4] Oct 30 19:03:25.786548 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.786490 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 19:03:25.792164 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.792057 2574 manager.go:217] Machine: {Timestamp:2026-10-30 19:03:25.790294765 +0000 UTC m=+0.378199933 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099266 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ad70a9a7e50c62fc39b52d44113a5 SystemUUID:ec2ad70a-9a7e-50c6-2fc3-9b52d44113a5 BootID:4f9335b3-918d-4333-a3f7-cd383854f7ad Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f7:77:57:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f7:77:57:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:a3:e6:80:90:88 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:03:25.792164 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.792163 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:03:25.792293 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.792281 2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:03:25.795796 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.795769 2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:03:25.795936 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.795797 2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-235.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:03:25.795983 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.795937 2574 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:03:25.795983 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.795944 2574 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:03:25.795983 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.795971 2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:03:25.796075 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.795984 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:03:25.796875 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.796865 2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:03:25.800186 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.800173 2574 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:03:25.800308 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.800298 2574 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:03:25.801143 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.801132 2574 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:03:25.801175 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.801166 2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:03:25.801207 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.801176 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:03:25.801207 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.801186 2574 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:03:25.801207 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.801198 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:03:25.801572 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.801541 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:03:25.804338 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.804321 2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:03:25.810086 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.810071 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:03:25.811059 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.811044 2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:03:25.811099 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.811083 2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:03:25.811293 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.811282 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:03:25.811322 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.811303 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:03:25.811322 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.811313 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:03:25.811389 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.811322 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:03:25.811389 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.811331 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:03:25.811389 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.811341 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:03:25.812229 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.812218 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:03:25.812264 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.812232 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:03:25.812264 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.812242 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:03:25.812264 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.812251 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:03:25.812344 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.812270 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:03:25.812344 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.812284 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:03:25.813058 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.813047 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:03:25.813089 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.813061 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:03:25.817605 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.817589 2574 server.go:1263] "Started kubelet" Oct 30 19:03:25.817707 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.817671 2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:03:25.820800 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.820739 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:03:25.820895 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.820829 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:03:25.821409 ip-10-0-140-235 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:03:25.824401 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.824382 2574 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:03:25.826020 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.825981 2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:03:25.828291 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:25.828272 2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 19:03:25.832160 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.832142 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:03:25.832641 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.832582 2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:03:25.832726 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:25.832694 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-235.ec2.internal\" not found" Oct 30 19:03:25.833455 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.833434 2574 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:03:25.833455 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.833455 2574 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:03:25.833597 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.833499 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:03:25.833597 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.833548 2574 factory.go:55] Registering systemd factory Oct 30 19:03:25.833597 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.833563 2574 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:03:25.833597 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.833568 2574 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:03:25.833597 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.833574 2574 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:03:25.833792 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.833755 2574 factory.go:153] Registering CRI-O factory Oct 30 19:03:25.833792 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.833762 2574 factory.go:223] Registration of the crio container factory successfully Oct 30 19:03:25.833862 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.833807 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:03:25.833862 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.833821 2574 factory.go:103] Registering Raw factory Oct 30 19:03:25.833862 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.833833 2574 manager.go:1199] Started watching for new ooms in manager Oct 30 19:03:25.834858 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.834843 2574 manager.go:319] Starting recovery of all containers Oct 30 19:03:25.844945 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.844929 2574 manager.go:324] Recovery completed Oct 30 19:03:25.849351 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.849334 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:03:25.851241 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.851225 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:03:25.851342 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.851258 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:03:25.851342 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.851273 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:03:25.851783 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.851767 2574 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:03:25.851783 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.851778 2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:03:25.851894 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.851796 2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:03:25.851894 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.851798 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:03:25.852761 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.852748 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:03:25.852822 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.852777 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:03:25.852822 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.852790 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:03:25.853339 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.853316 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:03:25.853404 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.853325 2574 policy_none.go:50] "Start" Oct 30 19:03:25.853404 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.853372 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:03:25.853404 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.853389 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:03:25.853549 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.853527 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-235.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 19:03:25.853628 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:25.853530 2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-235.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 19:03:25.854870 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:25.853544 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-235.ec2.internal.18e363fae28e8dcb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-235.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-235.ec2.internal,},FirstTimestamp:2026-10-30 19:03:25.817556427 +0000 UTC m=+0.405461600,LastTimestamp:2026-10-30 19:03:25.817556427 +0000 UTC m=+0.405461600,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-235.ec2.internal,}" Oct 30 19:03:25.855122 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.855086 2574 policy_none.go:44] "Start" Oct 30 19:03:25.869973 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.869950 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wlbnb" Oct 30 19:03:25.875500 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:25.875397 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-235.ec2.internal.18e363fae4908a0e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-235.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-235.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-235.ec2.internal,},FirstTimestamp:2026-10-30 19:03:25.851240974 +0000 UTC m=+0.439146144,LastTimestamp:2026-10-30 19:03:25.851240974 +0000 UTC m=+0.439146144,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-235.ec2.internal,}" Oct 30 19:03:25.875888 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:25.875860 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 19:03:25.875888 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:25.875860 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-235.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 19:03:25.877114 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.877097 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wlbnb" Oct 30 19:03:25.890476 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:25.890446 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 19:03:25.890562 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:25.890444 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-235.ec2.internal.18e363fae490e7f1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-235.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-235.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-235.ec2.internal,},FirstTimestamp:2026-10-30 19:03:25.851265009 +0000 UTC m=+0.439170179,LastTimestamp:2026-10-30 19:03:25.851265009 +0000 UTC m=+0.439170179,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-235.ec2.internal,}" Oct 30 19:03:25.933218 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:25.933195 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-235.ec2.internal\" not found" Oct 30 19:03:25.965644 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.965617 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:03:25.966879 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.966860 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:03:25.966879 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.966881 2574 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:03:25.967000 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.966902 2574 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:03:25.967000 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:25.966935 2574 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 19:03:25.982172 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.982155 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:03:25.993706 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.993654 2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:03:25.995954 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.995938 2574 manager.go:341] "Starting Device Plugin manager" Oct 30 19:03:25.996459 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:25.996445 2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:03:25.996508 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.996461 2574 server.go:87] "Starting device plugin registration server" Oct 30 19:03:25.996734 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.996724 2574 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:03:25.996767 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.996736 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:03:25.996894 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.996859 2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:03:25.997038 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.996990 2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:03:25.997038 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:25.997020 2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:03:25.997489 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:25.997467 2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:03:25.997562 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:25.997514 2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-235.ec2.internal\" not found" Oct 30 19:03:26.068545 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.068505 2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal"] Oct 30 19:03:26.068739 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.068596 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:03:26.069651 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.069634 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:03:26.069742 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.069671 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:03:26.069742 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.069686 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:03:26.071105 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.071089 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:03:26.071253 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.071238 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.071300 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.071269 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:03:26.072117 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.072100 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:03:26.072196 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.072131 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:03:26.072196 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.072146 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:03:26.072196 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.072159 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:03:26.072196 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.072185 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:03:26.072378 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.072199 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:03:26.073469 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.073456 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.073522 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.073480 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:03:26.074412 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.074396 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:03:26.074508 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.074423 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:03:26.074508 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.074437 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:03:26.083284 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:26.083267 2574 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-235.ec2.internal\" not found" node="ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.097467 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.097448 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:03:26.098415 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.098396 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:03:26.098489 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.098434 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:03:26.098489 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.098454 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:03:26.098489 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.098486 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.098965 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:26.098951 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-235.ec2.internal\" not found" node="ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.103193 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:26.103177 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-235.ec2.internal\" not found" node="ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.117283 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.117261 2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.117283 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:26.117284 2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-235.ec2.internal\": node \"ip-10-0-140-235.ec2.internal\" not found" Oct 30 19:03:26.134754 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.134733 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3aea26bca5b317ebce58e955350cfdd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal\" (UID: \"e3aea26bca5b317ebce58e955350cfdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.134842 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.134759 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3aea26bca5b317ebce58e955350cfdd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal\" (UID: \"e3aea26bca5b317ebce58e955350cfdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.156246 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:26.156223 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-235.ec2.internal\" not found" Oct 30 19:03:26.235376 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.235347 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3aea26bca5b317ebce58e955350cfdd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal\" (UID: \"e3aea26bca5b317ebce58e955350cfdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.235376 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.235353 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3aea26bca5b317ebce58e955350cfdd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal\" (UID: \"e3aea26bca5b317ebce58e955350cfdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.235511 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.235404 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b021a22e7072822cf09556c5f1f04826-config\") pod \"kube-apiserver-proxy-ip-10-0-140-235.ec2.internal\" (UID: \"b021a22e7072822cf09556c5f1f04826\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.235511 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.235424 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3aea26bca5b317ebce58e955350cfdd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal\" (UID: \"e3aea26bca5b317ebce58e955350cfdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.235511 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.235460 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3aea26bca5b317ebce58e955350cfdd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal\" (UID: \"e3aea26bca5b317ebce58e955350cfdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.256987 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:26.256934 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-235.ec2.internal\" not found" Oct 30 19:03:26.336534 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.336505 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b021a22e7072822cf09556c5f1f04826-config\") pod \"kube-apiserver-proxy-ip-10-0-140-235.ec2.internal\" (UID: \"b021a22e7072822cf09556c5f1f04826\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.336611 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.336564 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b021a22e7072822cf09556c5f1f04826-config\") pod \"kube-apiserver-proxy-ip-10-0-140-235.ec2.internal\" (UID: \"b021a22e7072822cf09556c5f1f04826\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.358062 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:26.358037 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-235.ec2.internal\" not found" Oct 30 19:03:26.404858 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.404823 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.407167 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.407150 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" Oct 30 19:03:26.460104 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:26.459894 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-235.ec2.internal\" not found" Oct 30 19:03:26.561358 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:26.561270 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-235.ec2.internal\" not found" Oct 30 19:03:26.662442 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:26.662399 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-235.ec2.internal\" not found" Oct 30 19:03:26.761166 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.761135 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 19:03:26.761654 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.761274 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:03:26.763452 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:26.763437 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-235.ec2.internal\" not found" Oct 30 19:03:26.864616 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:26.864535 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-235.ec2.internal\" not found" Oct 30 19:03:26.881730 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.881681 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:58:26 +0000 UTC" deadline="2028-04-07 14:39:56.446427308 +0000 UTC" Oct 30 19:03:26.881893 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.881714 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12595h36m29.564716868s" Oct 30 19:03:26.931340 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:26.931309 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:03:26.965325 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:26.965288 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-235.ec2.internal\" not found" Oct 30 19:03:27.052655 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:27.052610 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3aea26bca5b317ebce58e955350cfdd.slice/crio-11beabe9c6de3cd56340ccae3df3b8dae40820c3bc49d480260941265241cbdd WatchSource:0}: Error finding container 11beabe9c6de3cd56340ccae3df3b8dae40820c3bc49d480260941265241cbdd: Status 404 returned error can't find the container with id 11beabe9c6de3cd56340ccae3df3b8dae40820c3bc49d480260941265241cbdd Oct 30 19:03:27.053334 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:27.053305 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb021a22e7072822cf09556c5f1f04826.slice/crio-e5d5df8b1241cedec650cf6a47103044b6c2ea856b1f043f15fdd81ec60666f0 WatchSource:0}: Error finding container e5d5df8b1241cedec650cf6a47103044b6c2ea856b1f043f15fdd81ec60666f0: Status 404 returned error can't find the container with id e5d5df8b1241cedec650cf6a47103044b6c2ea856b1f043f15fdd81ec60666f0 Oct 30 19:03:27.058261 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.058238 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:03:27.065681 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:27.065657 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-235.ec2.internal\" not found" Oct 30 19:03:27.067997 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.067979 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:03:27.068894 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.068877 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:03:27.068969 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.068906 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:03:27.068969 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.068922 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:03:27.162978 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.162888 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:03:27.237119 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.237085 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:03:27.260993 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.260955 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:03:27.263199 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.263177 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" Oct 30 19:03:27.282316 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.282285 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:03:27.795845 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.795811 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:03:27.804964 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.804940 2574 apiserver.go:52] "Watching apiserver" Oct 30 19:03:27.829899 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.829849 2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:03:27.830361 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.830326 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-frj4v","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn","openshift-cluster-node-tuning-operator/tuned-bdc45","openshift-network-operator/iptables-alerter-wk24l","kube-system/global-pull-secret-syncer-hx495","kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal","openshift-network-diagnostics/network-check-target-zwhqt","openshift-dns/node-resolver-xpxgd","openshift-image-registry/node-ca-tbttr","openshift-multus/multus-kgvdt","openshift-ovn-kubernetes/ovnkube-node-7qftp","kube-system/konnectivity-agent-d26nk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal","openshift-multus/multus-additional-cni-plugins-k28t4"] Oct 30 19:03:27.833019 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.832978 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:27.833134 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:27.833097 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhqt" podUID="ee4d07b7-2acc-456e-932b-bfefb2f666df" Oct 30 19:03:27.833359 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.833268 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 19:03:27.835588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.835563 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.838979 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.838159 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:27.840315 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.840287 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:03:27.842672 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.842650 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:27.842772 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:27.842743 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx495" podUID="ad4cd78b-b491-4927-80ae-dae026d10e7a" Oct 30 19:03:27.845490 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.845469 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:27.846729 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.846703 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-kubelet\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.846813 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.846763 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.846813 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.846791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-ovnkube-config\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.846882 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.846818 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-env-overrides\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.846882 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.846846 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-systemd-units\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.846882 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.846871 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-log-socket\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.846972 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.846913 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-cni-bin\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.846972 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.846928 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-cni-netd\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.846972 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.846946 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.847090 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.846975 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftrxn\" (UniqueName: \"kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn\") pod \"network-check-target-zwhqt\" (UID: \"ee4d07b7-2acc-456e-932b-bfefb2f666df\") " pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:27.847090 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.846991 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-slash\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.847090 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.847008 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-run-netns\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.847090 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.847023 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-systemd\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.847090 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.847037 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-node-log\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.847090 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.847052 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-run-ovn-kubernetes\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.847090 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.847080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7537906-9d0e-4b31-9d76-45936780d55c-ovn-node-metrics-cert\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.847327 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.847103 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-ovnkube-script-lib\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.847327 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.847130 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-var-lib-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.847327 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.847163 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-etc-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.847327 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.847191 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-ovn\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.847327 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.847206 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5xn9\" (UniqueName: \"kubernetes.io/projected/f7537906-9d0e-4b31-9d76-45936780d55c-kube-api-access-r5xn9\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.847962 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.847939 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:03:27.850222 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.850204 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:03:27.851017 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.850986 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:03:27.851330 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.851314 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d26nk" Oct 30 19:03:27.851817 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.851795 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lgjqt\"" Oct 30 19:03:27.851915 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.851829 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:03:27.851915 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.851830 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fn8hc\"" Oct 30 19:03:27.852634 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.852600 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bwhgk\"" Oct 30 19:03:27.854144 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.854119 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:27.856529 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.856498 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kgvdt" Oct 30 19:03:27.858542 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.858521 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gp5x6\"" Oct 30 19:03:27.861282 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.861129 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:27.861282 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:27.861220 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frj4v" podUID="93f0a70e-357e-4d65-b554-42363280bebd" Oct 30 19:03:27.861437 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.861322 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vxbdc\"" Oct 30 19:03:27.863865 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.863841 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:03:27.875941 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.875916 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:03:27.876062 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.875986 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4szq8" Oct 30 19:03:27.885613 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.885590 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4szq8" Oct 30 19:03:27.886936 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.886902 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:03:27.934994 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.934967 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:03:27.948394 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948358 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-cnibin\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:27.948394 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948397 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-kubelet\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:27.948639 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948420 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-env-overrides\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.948639 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948435 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-host\") pod \"node-ca-tbttr\" (UID: \"830ddb4e-a8f8-406c-9e89-755bb0d48e0e\") " pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:03:27.948639 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948519 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-etc-selinux\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:27.948639 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948560 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-conf-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:27.948639 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948581 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-etc-kubernetes\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:27.948639 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948607 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-host\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:27.948639 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948638 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brtql\" (UniqueName: \"kubernetes.io/projected/d3c525ed-5017-482f-ac5c-670002363472-kube-api-access-brtql\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:27.948942 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-cni-netd\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.948942 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948707 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:27.948942 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948731 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-lib-modules\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:27.948942 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948753 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caef3ef2-f9af-4421-866e-61cc4febba1a-tmp\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:27.948942 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948780 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-slash\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.948942 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948805 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/633f6b63-a067-470b-bec9-e913e7f28589-agent-certs\") pod \"konnectivity-agent-d26nk\" (UID: \"633f6b63-a067-470b-bec9-e913e7f28589\") " pod="kube-system/konnectivity-agent-d26nk" Oct 30 19:03:27.948942 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-device-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:27.948942 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948896 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-kubernetes\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:27.949294 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948947 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-slash\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.949294 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysctl-d\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:27.949294 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.948991 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5xn9\" (UniqueName: \"kubernetes.io/projected/f7537906-9d0e-4b31-9d76-45936780d55c-kube-api-access-r5xn9\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.949294 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949043 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-cni-netd\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.949294 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949085 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-serviceca\") pod \"node-ca-tbttr\" (UID: \"830ddb4e-a8f8-406c-9e89-755bb0d48e0e\") " pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:03:27.949294 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949209 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:27.949294 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949239 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-socket-dir-parent\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:27.949294 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949281 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:27.949646 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949329 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/415a1d40-3275-4624-98cd-3df86bbc819a-tmp-dir\") pod \"node-resolver-xpxgd\" (UID: \"415a1d40-3275-4624-98cd-3df86bbc819a\") " pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:03:27.949646 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949353 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-system-cni-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:27.949646 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-cni-multus\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:27.949646 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949427 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysconfig\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:27.949646 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949447 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.949646 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949464 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slnzw\" (UniqueName: \"kubernetes.io/projected/415a1d40-3275-4624-98cd-3df86bbc819a-kube-api-access-slnzw\") pod \"node-resolver-xpxgd\" (UID: \"415a1d40-3275-4624-98cd-3df86bbc819a\") " pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:03:27.949646 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949461 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-env-overrides\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.949646 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949481 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-socket-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:27.949646 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949528 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-hostroot\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:27.949646 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949545 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm9bn\" (UniqueName: \"kubernetes.io/projected/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-kube-api-access-qm9bn\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:27.949646 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949561 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.949646 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949596 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-run-netns\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.949646 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949624 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-node-log\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.950067 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949660 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-run-netns\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.950067 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949702 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-cni-bin\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:27.950067 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949717 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-node-log\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.950067 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949769 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cnibin\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:27.950067 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad4cd78b-b491-4927-80ae-dae026d10e7a-dbus\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:27.950067 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949834 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-k8s-cni-cncf-io\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:27.950067 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949875 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-systemd\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:27.950067 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949905 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58fsk\" (UniqueName: \"kubernetes.io/projected/caef3ef2-f9af-4421-866e-61cc4febba1a-kube-api-access-58fsk\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:27.950067 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949925 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-ovnkube-config\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.950067 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:27.950067 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949978 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stb92\" (UniqueName: \"kubernetes.io/projected/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-kube-api-access-stb92\") pod \"node-ca-tbttr\" (UID: \"830ddb4e-a8f8-406c-9e89-755bb0d48e0e\") " pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:03:27.950067 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.949993 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-daemon-config\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:27.950067 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950010 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-tuned\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:27.950067 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950046 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-cni-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:27.950553 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950072 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-cni-binary-copy\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:27.950553 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950109 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:27.950553 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950253 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-cni-bin\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.950553 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950299 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrxn\" (UniqueName: \"kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn\") pod \"network-check-target-zwhqt\" (UID: \"ee4d07b7-2acc-456e-932b-bfefb2f666df\") " pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:27.950553 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950347 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cni-binary-copy\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:27.950553 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950409 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4mmt\" (UniqueName: \"kubernetes.io/projected/93f0a70e-357e-4d65-b554-42363280bebd-kube-api-access-m4mmt\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:27.950553 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950443 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-systemd\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.950553 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950477 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7537906-9d0e-4b31-9d76-45936780d55c-ovn-node-metrics-cert\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.950553 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950525 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/633f6b63-a067-470b-bec9-e913e7f28589-konnectivity-ca\") pod \"konnectivity-agent-d26nk\" (UID: \"633f6b63-a067-470b-bec9-e913e7f28589\") " pod="kube-system/konnectivity-agent-d26nk" Oct 30 19:03:27.950842 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950553 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-sys-fs\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:27.950842 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950605 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-netns\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:27.950842 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950663 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-sys\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:27.950842 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950691 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9r9b\" (UniqueName: \"kubernetes.io/projected/28f40fee-93d5-47de-89aa-943fa6b4efe7-kube-api-access-q9r9b\") pod \"iptables-alerter-wk24l\" (UID: \"28f40fee-93d5-47de-89aa-943fa6b4efe7\") " pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:03:27.950842 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950731 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-multus-certs\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:27.950842 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950756 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-run\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:27.950842 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950783 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28f40fee-93d5-47de-89aa-943fa6b4efe7-host-slash\") pod \"iptables-alerter-wk24l\" (UID: \"28f40fee-93d5-47de-89aa-943fa6b4efe7\") " pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:03:27.950842 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950816 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-kubelet\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951162 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950888 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951162 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950860 2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:03:27.951162 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950930 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-system-cni-dir\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:27.951162 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950958 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-modprobe-d\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:27.951162 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950975 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-systemd-units\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951162 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.950999 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-log-socket\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951162 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951023 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:27.951162 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951040 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad4cd78b-b491-4927-80ae-dae026d10e7a-kubelet-config\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:27.951162 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951072 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-registration-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:27.951162 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951099 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28f40fee-93d5-47de-89aa-943fa6b4efe7-iptables-alerter-script\") pod \"iptables-alerter-wk24l\" (UID: \"28f40fee-93d5-47de-89aa-943fa6b4efe7\") " pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:03:27.951162 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951123 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-cni-bin\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951162 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951127 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-run-ovn-kubernetes\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951176 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-run-ovn-kubernetes\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951244 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-ovnkube-script-lib\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951252 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-kubelet\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951273 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysctl-conf\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:27.951588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951292 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-systemd\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951303 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-var-lib-kubelet\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:27.951588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951308 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-systemd-units\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951361 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-log-socket\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951369 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-var-lib-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951431 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-etc-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951473 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-ovn\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951509 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-etc-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951561 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-ovn\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.951588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951564 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9gkl\" (UniqueName: \"kubernetes.io/projected/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-kube-api-access-p9gkl\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:27.951588 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951596 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-var-lib-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.952291 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.951629 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/415a1d40-3275-4624-98cd-3df86bbc819a-hosts-file\") pod \"node-resolver-xpxgd\" (UID: \"415a1d40-3275-4624-98cd-3df86bbc819a\") " pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:03:27.956030 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.955043 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7537906-9d0e-4b31-9d76-45936780d55c-ovn-node-metrics-cert\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.959376 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.959340 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:03:27.959486 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.959379 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:03:27.959525 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.959516 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:03:27.959765 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.959751 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:03:27.959810 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.959796 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:03:27.959909 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.959895 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:03:27.959957 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.959945 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:03:27.959957 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.959951 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:03:27.960017 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.959976 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:03:27.960059 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.960020 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:03:27.960059 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.960032 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:03:27.960166 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.960092 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:03:27.960166 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.960118 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:03:27.960166 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.960147 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:03:27.960166 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.960094 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:03:27.960317 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.960168 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:03:27.960317 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.960205 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:03:27.960715 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.960697 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-ovnkube-config\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.960863 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.960847 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:03:27.960940 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.960863 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:03:27.960998 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.960970 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:03:27.961150 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.961132 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:03:27.961239 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.961223 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:03:27.961269 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.961141 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2shm8\"" Oct 30 19:03:27.961454 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.961430 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:03:27.961454 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.961437 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j7nbz\"" Oct 30 19:03:27.961535 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.961521 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:03:27.961615 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.961603 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4kxxg\"" Oct 30 19:03:27.961664 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.961654 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9hj6l\"" Oct 30 19:03:27.962046 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.962028 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-ovnkube-script-lib\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.964465 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:27.964446 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:03:27.964528 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:27.964493 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:03:27.964528 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:27.964509 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ftrxn for pod openshift-network-diagnostics/network-check-target-zwhqt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:27.964626 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:27.964590 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn podName:ee4d07b7-2acc-456e-932b-bfefb2f666df nodeName:}" failed. No retries permitted until 2026-10-30 19:03:28.464558537 +0000 UTC m=+3.052463698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ftrxn" (UniqueName: "kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn") pod "network-check-target-zwhqt" (UID: "ee4d07b7-2acc-456e-932b-bfefb2f666df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:27.967192 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.967168 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5xn9\" (UniqueName: \"kubernetes.io/projected/f7537906-9d0e-4b31-9d76-45936780d55c-kube-api-access-r5xn9\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:27.972707 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.972665 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" event={"ID":"e3aea26bca5b317ebce58e955350cfdd","Type":"ContainerStarted","Data":"11beabe9c6de3cd56340ccae3df3b8dae40820c3bc49d480260941265241cbdd"} Oct 30 19:03:27.973980 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:27.973960 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" event={"ID":"b021a22e7072822cf09556c5f1f04826","Type":"ContainerStarted","Data":"e5d5df8b1241cedec650cf6a47103044b6c2ea856b1f043f15fdd81ec60666f0"} Oct 30 19:03:28.052390 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052262 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:28.052390 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052312 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stb92\" (UniqueName: \"kubernetes.io/projected/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-kube-api-access-stb92\") pod \"node-ca-tbttr\" (UID: \"830ddb4e-a8f8-406c-9e89-755bb0d48e0e\") " pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:03:28.052390 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-daemon-config\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.052390 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052362 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-tuned\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.052390 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-cni-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.052759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-cni-binary-copy\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.052759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:28.052759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052452 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cni-binary-copy\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:28.052759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052478 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mmt\" (UniqueName: \"kubernetes.io/projected/93f0a70e-357e-4d65-b554-42363280bebd-kube-api-access-m4mmt\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:28.052759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052508 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/633f6b63-a067-470b-bec9-e913e7f28589-konnectivity-ca\") pod \"konnectivity-agent-d26nk\" (UID: \"633f6b63-a067-470b-bec9-e913e7f28589\") " pod="kube-system/konnectivity-agent-d26nk" Oct 30 19:03:28.052759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-sys-fs\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:28.052759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052544 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-netns\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.052759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052558 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-sys\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.052759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052578 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9r9b\" (UniqueName: \"kubernetes.io/projected/28f40fee-93d5-47de-89aa-943fa6b4efe7-kube-api-access-q9r9b\") pod \"iptables-alerter-wk24l\" (UID: \"28f40fee-93d5-47de-89aa-943fa6b4efe7\") " pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:03:28.052759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-multus-certs\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.052759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052611 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-run\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.052759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052633 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28f40fee-93d5-47de-89aa-943fa6b4efe7-host-slash\") pod \"iptables-alerter-wk24l\" (UID: \"28f40fee-93d5-47de-89aa-943fa6b4efe7\") " pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:03:28.052759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052656 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-system-cni-dir\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:28.052759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052677 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-modprobe-d\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.052759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052695 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:28.052759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052712 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad4cd78b-b491-4927-80ae-dae026d10e7a-kubelet-config\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:28.052759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052733 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-registration-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:28.053442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052756 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28f40fee-93d5-47de-89aa-943fa6b4efe7-iptables-alerter-script\") pod \"iptables-alerter-wk24l\" (UID: \"28f40fee-93d5-47de-89aa-943fa6b4efe7\") " pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:03:28.053442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052783 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysctl-conf\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.053442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052798 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-var-lib-kubelet\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.053442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052827 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9gkl\" (UniqueName: \"kubernetes.io/projected/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-kube-api-access-p9gkl\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:28.053442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/415a1d40-3275-4624-98cd-3df86bbc819a-hosts-file\") pod \"node-resolver-xpxgd\" (UID: \"415a1d40-3275-4624-98cd-3df86bbc819a\") " pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:03:28.053442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052869 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-cnibin\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.053442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-kubelet\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.053442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052915 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-host\") pod \"node-ca-tbttr\" (UID: \"830ddb4e-a8f8-406c-9e89-755bb0d48e0e\") " pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:03:28.053442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052950 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-etc-selinux\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:28.053442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052966 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-conf-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.053442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.052984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-etc-kubernetes\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.053442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-host\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.053442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brtql\" (UniqueName: \"kubernetes.io/projected/d3c525ed-5017-482f-ac5c-670002363472-kube-api-access-brtql\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:28.053442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:28.053442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053060 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-lib-modules\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.053442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053078 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caef3ef2-f9af-4421-866e-61cc4febba1a-tmp\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.053442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053120 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/633f6b63-a067-470b-bec9-e913e7f28589-agent-certs\") pod \"konnectivity-agent-d26nk\" (UID: \"633f6b63-a067-470b-bec9-e913e7f28589\") " pod="kube-system/konnectivity-agent-d26nk" Oct 30 19:03:28.054218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053146 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-device-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:28.054218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-kubernetes\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.054218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053215 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysctl-d\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.054218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053248 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-serviceca\") pod \"node-ca-tbttr\" (UID: \"830ddb4e-a8f8-406c-9e89-755bb0d48e0e\") " pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:03:28.054218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053276 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:28.054218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-socket-dir-parent\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.054218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053314 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:28.054218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/415a1d40-3275-4624-98cd-3df86bbc819a-tmp-dir\") pod \"node-resolver-xpxgd\" (UID: \"415a1d40-3275-4624-98cd-3df86bbc819a\") " pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:03:28.054218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053359 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-system-cni-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.054218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053379 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-cni-multus\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.054218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysconfig\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.054218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053412 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slnzw\" (UniqueName: \"kubernetes.io/projected/415a1d40-3275-4624-98cd-3df86bbc819a-kube-api-access-slnzw\") pod \"node-resolver-xpxgd\" (UID: \"415a1d40-3275-4624-98cd-3df86bbc819a\") " pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:03:28.054218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-socket-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:28.054218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053483 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-hostroot\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.054218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm9bn\" (UniqueName: \"kubernetes.io/projected/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-kube-api-access-qm9bn\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.054218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-cni-bin\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.054218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053577 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cnibin\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:28.054918 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053605 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad4cd78b-b491-4927-80ae-dae026d10e7a-dbus\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:28.054918 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053655 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-k8s-cni-cncf-io\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.054918 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053672 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-systemd\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.054918 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.053701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fsk\" (UniqueName: \"kubernetes.io/projected/caef3ef2-f9af-4421-866e-61cc4febba1a-kube-api-access-58fsk\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.054918 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:28.054129 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:28.054918 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:28.054193 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs podName:93f0a70e-357e-4d65-b554-42363280bebd nodeName:}" failed. No retries permitted until 2026-10-30 19:03:28.554179453 +0000 UTC m=+3.142084623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs") pod "network-metrics-daemon-frj4v" (UID: "93f0a70e-357e-4d65-b554-42363280bebd") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:28.055198 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.055071 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-daemon-config\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.055492 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.055462 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-conf-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.055575 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.055530 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-system-cni-dir\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:28.055636 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.055616 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-modprobe-d\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.055718 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.055688 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-cni-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.056243 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.056216 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:28.056243 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.056225 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-kubelet\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.056380 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.056296 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-host\") pod \"node-ca-tbttr\" (UID: \"830ddb4e-a8f8-406c-9e89-755bb0d48e0e\") " pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:03:28.056380 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.056324 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/415a1d40-3275-4624-98cd-3df86bbc819a-hosts-file\") pod \"node-resolver-xpxgd\" (UID: \"415a1d40-3275-4624-98cd-3df86bbc819a\") " pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:03:28.056477 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.056391 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-cnibin\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.056477 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.056414 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-etc-selinux\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:28.056477 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.056441 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad4cd78b-b491-4927-80ae-dae026d10e7a-kubelet-config\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:28.056648 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.056523 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-registration-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:28.057156 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.057123 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-hostroot\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.057245 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.057137 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-cni-multus\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.057245 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.057228 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysconfig\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.057464 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.057438 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:28.057596 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.057506 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-etc-kubernetes\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.057596 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.057560 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-host\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.057706 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.057600 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-socket-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:28.057706 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.057669 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-device-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:28.057823 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.057721 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:28.057823 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.057799 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-kubernetes\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.057930 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.057824 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-lib-modules\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.067224 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:28.057884 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:28.067224 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.057933 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-socket-dir-parent\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.067428 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:28.067238 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret podName:ad4cd78b-b491-4927-80ae-dae026d10e7a nodeName:}" failed. No retries permitted until 2026-10-30 19:03:28.56721356 +0000 UTC m=+3.155118732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret") pod "global-pull-secret-syncer-hx495" (UID: "ad4cd78b-b491-4927-80ae-dae026d10e7a") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:28.067428 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.057984 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-system-cni-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.067428 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.058214 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-serviceca\") pod \"node-ca-tbttr\" (UID: \"830ddb4e-a8f8-406c-9e89-755bb0d48e0e\") " pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:03:28.067428 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.058234 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-cni-bin\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.067428 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.058272 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cnibin\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:28.067428 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.058287 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysctl-conf\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.067428 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.058311 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysctl-d\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.067428 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.058334 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-netns\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.067428 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.058354 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-var-lib-kubelet\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.067428 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.058388 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-multus-certs\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.067428 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.058427 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-sys\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.067428 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.058473 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:28.067428 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.058484 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28f40fee-93d5-47de-89aa-943fa6b4efe7-iptables-alerter-script\") pod \"iptables-alerter-wk24l\" (UID: \"28f40fee-93d5-47de-89aa-943fa6b4efe7\") " pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:03:28.067428 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.058668 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-systemd\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.067428 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.058708 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-k8s-cni-cncf-io\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.067428 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.058938 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-sys-fs\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:28.067428 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.058981 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-run\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.067995 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.058980 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cni-binary-copy\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:28.067995 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.059022 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28f40fee-93d5-47de-89aa-943fa6b4efe7-host-slash\") pod \"iptables-alerter-wk24l\" (UID: \"28f40fee-93d5-47de-89aa-943fa6b4efe7\") " pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:03:28.067995 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.059066 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/633f6b63-a067-470b-bec9-e913e7f28589-konnectivity-ca\") pod \"konnectivity-agent-d26nk\" (UID: \"633f6b63-a067-470b-bec9-e913e7f28589\") " pod="kube-system/konnectivity-agent-d26nk" Oct 30 19:03:28.067995 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.059201 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-cni-binary-copy\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.067995 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.059414 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/415a1d40-3275-4624-98cd-3df86bbc819a-tmp-dir\") pod \"node-resolver-xpxgd\" (UID: \"415a1d40-3275-4624-98cd-3df86bbc819a\") " pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:03:28.067995 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.059592 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-tuned\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.067995 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.061012 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caef3ef2-f9af-4421-866e-61cc4febba1a-tmp\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.067995 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.061788 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/633f6b63-a067-470b-bec9-e913e7f28589-agent-certs\") pod \"konnectivity-agent-d26nk\" (UID: \"633f6b63-a067-470b-bec9-e913e7f28589\") " pod="kube-system/konnectivity-agent-d26nk" Oct 30 19:03:28.067995 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.058020 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad4cd78b-b491-4927-80ae-dae026d10e7a-dbus\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:28.075564 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.075530 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58fsk\" (UniqueName: \"kubernetes.io/projected/caef3ef2-f9af-4421-866e-61cc4febba1a-kube-api-access-58fsk\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.075970 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.075945 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stb92\" (UniqueName: \"kubernetes.io/projected/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-kube-api-access-stb92\") pod \"node-ca-tbttr\" (UID: \"830ddb4e-a8f8-406c-9e89-755bb0d48e0e\") " pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:03:28.079100 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.078627 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slnzw\" (UniqueName: \"kubernetes.io/projected/415a1d40-3275-4624-98cd-3df86bbc819a-kube-api-access-slnzw\") pod \"node-resolver-xpxgd\" (UID: \"415a1d40-3275-4624-98cd-3df86bbc819a\") " pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:03:28.079100 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.078657 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brtql\" (UniqueName: \"kubernetes.io/projected/d3c525ed-5017-482f-ac5c-670002363472-kube-api-access-brtql\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:28.079100 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.078631 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4mmt\" (UniqueName: \"kubernetes.io/projected/93f0a70e-357e-4d65-b554-42363280bebd-kube-api-access-m4mmt\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:28.079100 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.079023 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm9bn\" (UniqueName: \"kubernetes.io/projected/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-kube-api-access-qm9bn\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.079359 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.079184 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9gkl\" (UniqueName: \"kubernetes.io/projected/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-kube-api-access-p9gkl\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:28.079911 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.079886 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9r9b\" (UniqueName: \"kubernetes.io/projected/28f40fee-93d5-47de-89aa-943fa6b4efe7-kube-api-access-q9r9b\") pod \"iptables-alerter-wk24l\" (UID: \"28f40fee-93d5-47de-89aa-943fa6b4efe7\") " pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:03:28.167647 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.167604 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:03:28.169947 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.169916 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:03:28.176411 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.176384 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:28.178808 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:28.178780 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaef3ef2_f9af_4421_866e_61cc4febba1a.slice/crio-ddf56d93f0c4f518b9d8e1714094e23b525c3b345d127ab2dbad5360495d61d7 WatchSource:0}: Error finding container ddf56d93f0c4f518b9d8e1714094e23b525c3b345d127ab2dbad5360495d61d7: Status 404 returned error can't find the container with id ddf56d93f0c4f518b9d8e1714094e23b525c3b345d127ab2dbad5360495d61d7 Oct 30 19:03:28.181348 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:28.181321 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f40fee_93d5_47de_89aa_943fa6b4efe7.slice/crio-bcb44cd7dc162cffc616356e2a7bd5e54c05396519e79f2008d54d2a075f51ad WatchSource:0}: Error finding container bcb44cd7dc162cffc616356e2a7bd5e54c05396519e79f2008d54d2a075f51ad: Status 404 returned error can't find the container with id bcb44cd7dc162cffc616356e2a7bd5e54c05396519e79f2008d54d2a075f51ad Oct 30 19:03:28.187555 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:28.187530 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7537906_9d0e_4b31_9d76_45936780d55c.slice/crio-fad4b0bf404d53e190cb2aa2851957981a46518e308a53dc7a31ef760e520314 WatchSource:0}: Error finding container fad4b0bf404d53e190cb2aa2851957981a46518e308a53dc7a31ef760e520314: Status 404 returned error can't find the container with id fad4b0bf404d53e190cb2aa2851957981a46518e308a53dc7a31ef760e520314 Oct 30 19:03:28.192121 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.192067 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:03:28.195393 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.195357 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:03:28.199871 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.199847 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d26nk" Oct 30 19:03:28.206332 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:28.206301 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe5ab76_2e1c_4dda_bfb8_6bf9c48dccad.slice/crio-7fe8396ac1bb3f74a1e74c4c62f25b12e0eb09e446a3712167e48e0e6154dec4 WatchSource:0}: Error finding container 7fe8396ac1bb3f74a1e74c4c62f25b12e0eb09e446a3712167e48e0e6154dec4: Status 404 returned error can't find the container with id 7fe8396ac1bb3f74a1e74c4c62f25b12e0eb09e446a3712167e48e0e6154dec4 Oct 30 19:03:28.210074 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:28.210047 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415a1d40_3275_4624_98cd_3df86bbc819a.slice/crio-287bd230c933af1b79430cbd50c3f852bdf540f15b5b12a8df5b7853869af3fc WatchSource:0}: Error finding container 287bd230c933af1b79430cbd50c3f852bdf540f15b5b12a8df5b7853869af3fc: Status 404 returned error can't find the container with id 287bd230c933af1b79430cbd50c3f852bdf540f15b5b12a8df5b7853869af3fc Oct 30 19:03:28.213169 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.213146 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:03:28.214011 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:28.213990 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633f6b63_a067_470b_bec9_e913e7f28589.slice/crio-1ab31aab904cf00ea369247612bac2a6903a09991025c519e691a647ecab581f WatchSource:0}: Error finding container 1ab31aab904cf00ea369247612bac2a6903a09991025c519e691a647ecab581f: Status 404 returned error can't find the container with id 1ab31aab904cf00ea369247612bac2a6903a09991025c519e691a647ecab581f Oct 30 19:03:28.216995 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.216974 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kgvdt" Oct 30 19:03:28.223884 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.223864 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:03:28.226685 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:28.226647 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c525ed_5017_482f_ac5c_670002363472.slice/crio-69b1e146aaabdee04685a2656258c11196f9e39a0605b2629b4fd0732be785c3 WatchSource:0}: Error finding container 69b1e146aaabdee04685a2656258c11196f9e39a0605b2629b4fd0732be785c3: Status 404 returned error can't find the container with id 69b1e146aaabdee04685a2656258c11196f9e39a0605b2629b4fd0732be785c3 Oct 30 19:03:28.242221 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:28.242185 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830ddb4e_a8f8_406c_9e89_755bb0d48e0e.slice/crio-91fc483e041346371d853e49b581b082802ddff668b1d2893b35150ba16c0aa2 WatchSource:0}: Error finding container 91fc483e041346371d853e49b581b082802ddff668b1d2893b35150ba16c0aa2: Status 404 returned error can't find the container with id 91fc483e041346371d853e49b581b082802ddff668b1d2893b35150ba16c0aa2 Oct 30 19:03:28.558543 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.558506 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:28.558543 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.558548 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrxn\" (UniqueName: \"kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn\") pod \"network-check-target-zwhqt\" (UID: \"ee4d07b7-2acc-456e-932b-bfefb2f666df\") " pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:28.558783 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:28.558677 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:03:28.558783 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:28.558681 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:28.558783 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:28.558765 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs podName:93f0a70e-357e-4d65-b554-42363280bebd nodeName:}" failed. No retries permitted until 2026-10-30 19:03:29.558743046 +0000 UTC m=+4.146648223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs") pod "network-metrics-daemon-frj4v" (UID: "93f0a70e-357e-4d65-b554-42363280bebd") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:28.558921 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:28.558690 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:03:28.558921 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:28.558807 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ftrxn for pod openshift-network-diagnostics/network-check-target-zwhqt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:28.558921 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:28.558865 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn podName:ee4d07b7-2acc-456e-932b-bfefb2f666df nodeName:}" failed. No retries permitted until 2026-10-30 19:03:29.55884722 +0000 UTC m=+4.146752396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ftrxn" (UniqueName: "kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn") pod "network-check-target-zwhqt" (UID: "ee4d07b7-2acc-456e-932b-bfefb2f666df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:28.660208 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.659423 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:28.660208 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:28.659609 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:28.660208 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:28.659768 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret podName:ad4cd78b-b491-4927-80ae-dae026d10e7a nodeName:}" failed. No retries permitted until 2026-10-30 19:03:29.659746521 +0000 UTC m=+4.247651699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret") pod "global-pull-secret-syncer-hx495" (UID: "ad4cd78b-b491-4927-80ae-dae026d10e7a") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:28.889876 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.889802 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:58:28 +0000 UTC" deadline="2028-04-18 16:58:36.58545221 +0000 UTC" Oct 30 19:03:28.889876 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.889874 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12861h55m7.695583443s" Oct 30 19:03:28.979871 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.979674 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d26nk" event={"ID":"633f6b63-a067-470b-bec9-e913e7f28589","Type":"ContainerStarted","Data":"1ab31aab904cf00ea369247612bac2a6903a09991025c519e691a647ecab581f"} Oct 30 19:03:28.983939 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.983892 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xpxgd" event={"ID":"415a1d40-3275-4624-98cd-3df86bbc819a","Type":"ContainerStarted","Data":"287bd230c933af1b79430cbd50c3f852bdf540f15b5b12a8df5b7853869af3fc"} Oct 30 19:03:28.993789 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:28.993743 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k28t4" event={"ID":"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad","Type":"ContainerStarted","Data":"7fe8396ac1bb3f74a1e74c4c62f25b12e0eb09e446a3712167e48e0e6154dec4"} Oct 30 19:03:29.001666 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:29.001627 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wk24l" event={"ID":"28f40fee-93d5-47de-89aa-943fa6b4efe7","Type":"ContainerStarted","Data":"bcb44cd7dc162cffc616356e2a7bd5e54c05396519e79f2008d54d2a075f51ad"} Oct 30 19:03:29.008597 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:29.008526 2574 generic.go:356] "Generic (PLEG): container finished" podID="e3aea26bca5b317ebce58e955350cfdd" containerID="50bacf6f4746f915de68ab49fc167fd35f3fe4d71a4e9d3ad03ecb007d8e3e17" exitCode=0 Oct 30 19:03:29.008766 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:29.008660 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" event={"ID":"e3aea26bca5b317ebce58e955350cfdd","Type":"ContainerDied","Data":"50bacf6f4746f915de68ab49fc167fd35f3fe4d71a4e9d3ad03ecb007d8e3e17"} Oct 30 19:03:29.016855 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:29.016817 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tbttr" event={"ID":"830ddb4e-a8f8-406c-9e89-755bb0d48e0e","Type":"ContainerStarted","Data":"91fc483e041346371d853e49b581b082802ddff668b1d2893b35150ba16c0aa2"} Oct 30 19:03:29.024626 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:29.024593 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kgvdt" event={"ID":"ceff794b-7638-49c7-a0f4-495a8cf3d8c7","Type":"ContainerStarted","Data":"ea757d4eaa177307762b1f6748e8cb48eccf56fd484c16a3bd72ed4378137412"} Oct 30 19:03:29.034184 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:29.034148 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" event={"ID":"d3c525ed-5017-482f-ac5c-670002363472","Type":"ContainerStarted","Data":"69b1e146aaabdee04685a2656258c11196f9e39a0605b2629b4fd0732be785c3"} Oct 30 19:03:29.036888 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:29.036855 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" event={"ID":"f7537906-9d0e-4b31-9d76-45936780d55c","Type":"ContainerStarted","Data":"fad4b0bf404d53e190cb2aa2851957981a46518e308a53dc7a31ef760e520314"} Oct 30 19:03:29.040012 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:29.039983 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bdc45" event={"ID":"caef3ef2-f9af-4421-866e-61cc4febba1a","Type":"ContainerStarted","Data":"ddf56d93f0c4f518b9d8e1714094e23b525c3b345d127ab2dbad5360495d61d7"} Oct 30 19:03:29.570127 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:29.567247 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:29.570127 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:29.567584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrxn\" (UniqueName: \"kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn\") pod \"network-check-target-zwhqt\" (UID: \"ee4d07b7-2acc-456e-932b-bfefb2f666df\") " pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:29.570127 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:29.567813 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:03:29.570127 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:29.567836 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:03:29.570127 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:29.567850 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ftrxn for pod openshift-network-diagnostics/network-check-target-zwhqt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:29.570127 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:29.567908 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn podName:ee4d07b7-2acc-456e-932b-bfefb2f666df nodeName:}" failed. No retries permitted until 2026-10-30 19:03:31.567888878 +0000 UTC m=+6.155794041 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ftrxn" (UniqueName: "kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn") pod "network-check-target-zwhqt" (UID: "ee4d07b7-2acc-456e-932b-bfefb2f666df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:29.570127 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:29.568367 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:29.570127 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:29.568428 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs podName:93f0a70e-357e-4d65-b554-42363280bebd nodeName:}" failed. No retries permitted until 2026-10-30 19:03:31.568410255 +0000 UTC m=+6.156315419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs") pod "network-metrics-daemon-frj4v" (UID: "93f0a70e-357e-4d65-b554-42363280bebd") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:29.668981 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:29.668943 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:29.669233 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:29.669152 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:29.669233 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:29.669222 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret podName:ad4cd78b-b491-4927-80ae-dae026d10e7a nodeName:}" failed. No retries permitted until 2026-10-30 19:03:31.669202486 +0000 UTC m=+6.257107659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret") pod "global-pull-secret-syncer-hx495" (UID: "ad4cd78b-b491-4927-80ae-dae026d10e7a") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:29.891239 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:29.891142 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:58:28 +0000 UTC" deadline="2028-06-10 08:46:25.558540748 +0000 UTC" Oct 30 19:03:29.891239 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:29.891188 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14125h42m55.667357782s" Oct 30 19:03:29.970766 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:29.970732 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:29.970952 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:29.970887 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frj4v" podUID="93f0a70e-357e-4d65-b554-42363280bebd" Oct 30 19:03:29.971354 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:29.971329 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:29.971454 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:29.971433 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhqt" podUID="ee4d07b7-2acc-456e-932b-bfefb2f666df" Oct 30 19:03:29.971516 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:29.971500 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:29.971588 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:29.971562 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx495" podUID="ad4cd78b-b491-4927-80ae-dae026d10e7a" Oct 30 19:03:30.052473 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:30.051589 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" event={"ID":"b021a22e7072822cf09556c5f1f04826","Type":"ContainerStarted","Data":"6296437556c50144bdb705d00cb452da46161093185cb1906e3c778ad189c014"} Oct 30 19:03:30.059097 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:30.058820 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" event={"ID":"e3aea26bca5b317ebce58e955350cfdd","Type":"ContainerStarted","Data":"890737c9f8c95876cdf4786dbd1fee98d3e378206b04a9495f6c50618318f563"} Oct 30 19:03:30.086332 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:30.086254 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" podStartSLOduration=3.0862343230000002 podStartE2EDuration="3.086234323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:30.085604258 +0000 UTC m=+4.673509439" watchObservedRunningTime="2026-10-30 19:03:30.086234323 +0000 UTC m=+4.674139505" Oct 30 19:03:31.585978 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:31.585886 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:31.585978 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:31.585945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrxn\" (UniqueName: \"kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn\") pod \"network-check-target-zwhqt\" (UID: \"ee4d07b7-2acc-456e-932b-bfefb2f666df\") " pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:31.586536 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:31.586133 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:03:31.586536 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:31.586158 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:03:31.586536 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:31.586174 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ftrxn for pod openshift-network-diagnostics/network-check-target-zwhqt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:31.586536 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:31.586242 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn podName:ee4d07b7-2acc-456e-932b-bfefb2f666df nodeName:}" failed. No retries permitted until 2026-10-30 19:03:35.586222232 +0000 UTC m=+10.174127406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ftrxn" (UniqueName: "kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn") pod "network-check-target-zwhqt" (UID: "ee4d07b7-2acc-456e-932b-bfefb2f666df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:31.586758 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:31.586692 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:31.586758 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:31.586751 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs podName:93f0a70e-357e-4d65-b554-42363280bebd nodeName:}" failed. No retries permitted until 2026-10-30 19:03:35.586731738 +0000 UTC m=+10.174636912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs") pod "network-metrics-daemon-frj4v" (UID: "93f0a70e-357e-4d65-b554-42363280bebd") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:31.686600 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:31.686560 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:31.686803 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:31.686745 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:31.686875 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:31.686813 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret podName:ad4cd78b-b491-4927-80ae-dae026d10e7a nodeName:}" failed. No retries permitted until 2026-10-30 19:03:35.686793821 +0000 UTC m=+10.274698993 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret") pod "global-pull-secret-syncer-hx495" (UID: "ad4cd78b-b491-4927-80ae-dae026d10e7a") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:31.970073 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:31.969296 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:31.970073 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:31.969456 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frj4v" podUID="93f0a70e-357e-4d65-b554-42363280bebd" Oct 30 19:03:31.970073 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:31.969878 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:31.970073 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:31.969974 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhqt" podUID="ee4d07b7-2acc-456e-932b-bfefb2f666df" Oct 30 19:03:31.970447 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:31.970421 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:31.970550 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:31.970531 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx495" podUID="ad4cd78b-b491-4927-80ae-dae026d10e7a" Oct 30 19:03:32.071064 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:32.070988 2574 generic.go:356] "Generic (PLEG): container finished" podID="7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad" containerID="e5c0078982f8695117aa7e623c90d2b8b13d0bec91e123831b9d9493d60c12ca" exitCode=0 Oct 30 19:03:32.071269 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:32.071086 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k28t4" event={"ID":"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad","Type":"ContainerDied","Data":"e5c0078982f8695117aa7e623c90d2b8b13d0bec91e123831b9d9493d60c12ca"} Oct 30 19:03:32.135499 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:32.134902 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" podStartSLOduration=5.134880566 podStartE2EDuration="5.134880566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:30.121133047 +0000 UTC m=+4.709038230" watchObservedRunningTime="2026-10-30 19:03:32.134880566 +0000 UTC m=+6.722785749" Oct 30 19:03:33.077197 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:33.077157 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d26nk" event={"ID":"633f6b63-a067-470b-bec9-e913e7f28589","Type":"ContainerStarted","Data":"e721ff2d277e6e9d07a7a0fc0882132664b4bda614819e09f0e15abfb2f4efb4"} Oct 30 19:03:33.969490 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:33.968127 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:33.969490 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:33.968253 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frj4v" podUID="93f0a70e-357e-4d65-b554-42363280bebd" Oct 30 19:03:33.969490 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:33.968581 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:33.969490 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:33.968678 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhqt" podUID="ee4d07b7-2acc-456e-932b-bfefb2f666df" Oct 30 19:03:33.969490 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:33.968729 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:33.969490 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:33.968794 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx495" podUID="ad4cd78b-b491-4927-80ae-dae026d10e7a" Oct 30 19:03:34.082305 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:34.082271 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tbttr" event={"ID":"830ddb4e-a8f8-406c-9e89-755bb0d48e0e","Type":"ContainerStarted","Data":"4525e6d2abacc9a8b2e8f6921095f8748a473c42fc682d4c8e130a642727c669"} Oct 30 19:03:34.115860 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:34.115801 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tbttr" podStartSLOduration=1.947017431 podStartE2EDuration="7.115779749s" totalImagesPullingTime="5.168762318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:34.115536929 +0000 UTC m=+8.703442108" watchObservedRunningTime="2026-10-30 19:03:34.115779749 +0000 UTC m=+8.703684927" Oct 30 19:03:34.116190 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:34.116147 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d26nk" podStartSLOduration=4.017186666 podStartE2EDuration="8.116132304s" totalImagesPullingTime="4.098945638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:33.110090979 +0000 UTC m=+7.697996250" watchObservedRunningTime="2026-10-30 19:03:34.116132304 +0000 UTC m=+8.704037483" Oct 30 19:03:35.089515 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:35.089481 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" event={"ID":"d3c525ed-5017-482f-ac5c-670002363472","Type":"ContainerStarted","Data":"32e9e37fd9a3f626f35e9af912b7595e0d5ab31d4e0f263dc1be53b7765c79c8"} Oct 30 19:03:35.621520 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:35.621482 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:35.621702 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:35.621540 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrxn\" (UniqueName: \"kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn\") pod \"network-check-target-zwhqt\" (UID: \"ee4d07b7-2acc-456e-932b-bfefb2f666df\") " pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:35.621775 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:35.621736 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:03:35.621775 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:35.621758 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:03:35.621775 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:35.621771 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ftrxn for pod openshift-network-diagnostics/network-check-target-zwhqt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:35.621932 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:35.621826 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn podName:ee4d07b7-2acc-456e-932b-bfefb2f666df nodeName:}" failed. No retries permitted until 2026-10-30 19:03:43.621809163 +0000 UTC m=+18.209714321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ftrxn" (UniqueName: "kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn") pod "network-check-target-zwhqt" (UID: "ee4d07b7-2acc-456e-932b-bfefb2f666df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:35.622277 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:35.622250 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:35.622352 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:35.622305 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs podName:93f0a70e-357e-4d65-b554-42363280bebd nodeName:}" failed. No retries permitted until 2026-10-30 19:03:43.622290524 +0000 UTC m=+18.210195693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs") pod "network-metrics-daemon-frj4v" (UID: "93f0a70e-357e-4d65-b554-42363280bebd") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:35.722726 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:35.722697 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:35.722856 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:35.722833 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:35.722904 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:35.722894 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret podName:ad4cd78b-b491-4927-80ae-dae026d10e7a nodeName:}" failed. No retries permitted until 2026-10-30 19:03:43.722875924 +0000 UTC m=+18.310781083 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret") pod "global-pull-secret-syncer-hx495" (UID: "ad4cd78b-b491-4927-80ae-dae026d10e7a") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:35.864385 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:35.864353 2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:03:35.968843 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:35.968796 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:35.969042 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:35.968951 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frj4v" podUID="93f0a70e-357e-4d65-b554-42363280bebd" Oct 30 19:03:35.969405 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:35.969380 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:35.969509 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:35.969481 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhqt" podUID="ee4d07b7-2acc-456e-932b-bfefb2f666df" Oct 30 19:03:35.969613 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:35.969593 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:35.969740 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:35.969681 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx495" podUID="ad4cd78b-b491-4927-80ae-dae026d10e7a" Oct 30 19:03:36.005386 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:36.004690 2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:03:35.864378422Z","UUID":"4cb7708b-9b21-43ac-9025-a0dc5ab34f43","Handler":null,"Name":"","Endpoint":""} Oct 30 19:03:36.008608 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:36.008586 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:03:36.008770 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:36.008760 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:03:36.094440 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:36.093890 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xpxgd" event={"ID":"415a1d40-3275-4624-98cd-3df86bbc819a","Type":"ContainerStarted","Data":"e270c402373696ba30ecfc6e2622b03fe699a4c83f791c00ccdf1f4918616d42"} Oct 30 19:03:36.097641 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:36.096926 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" event={"ID":"d3c525ed-5017-482f-ac5c-670002363472","Type":"ContainerStarted","Data":"7416eef1844260a70ada05714a13efe44e87774a4308300e8b3850a1800592ee"} Oct 30 19:03:37.102948 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:37.102043 2574 generic.go:356] "Generic (PLEG): container finished" podID="7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad" containerID="3ea4dac667a65d32a0b1107bde80031e155b3580cd78a85983b34148206194b8" exitCode=0 Oct 30 19:03:37.102948 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:37.102140 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k28t4" event={"ID":"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad","Type":"ContainerDied","Data":"3ea4dac667a65d32a0b1107bde80031e155b3580cd78a85983b34148206194b8"} Oct 30 19:03:37.108536 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:37.108369 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kgvdt" event={"ID":"ceff794b-7638-49c7-a0f4-495a8cf3d8c7","Type":"ContainerStarted","Data":"c6ce3fa9d03dfe6b2332916e5f8381690f7ca8d0bc4c129d2e334b1c394f6c72"} Oct 30 19:03:37.149730 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:37.149674 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-xpxgd" podStartSLOduration=3.476826543 podStartE2EDuration="11.149654836s" totalImagesPullingTime="7.672828293s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:36.125966294 +0000 UTC m=+10.713871476" watchObservedRunningTime="2026-10-30 19:03:37.149654836 +0000 UTC m=+11.737560015" Oct 30 19:03:37.181959 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:37.181903 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-kgvdt" podStartSLOduration=3.366953628 podStartE2EDuration="11.181890652s" totalImagesPullingTime="7.814937024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:37.181835638 +0000 UTC m=+11.769740817" watchObservedRunningTime="2026-10-30 19:03:37.181890652 +0000 UTC m=+11.769795829" Oct 30 19:03:37.693682 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:37.693414 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d26nk" Oct 30 19:03:37.694324 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:37.694107 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d26nk" Oct 30 19:03:37.968197 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:37.968118 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:37.968338 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:37.968121 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:37.968338 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:37.968252 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:37.968338 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:37.968258 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frj4v" podUID="93f0a70e-357e-4d65-b554-42363280bebd" Oct 30 19:03:37.968487 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:37.968356 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx495" podUID="ad4cd78b-b491-4927-80ae-dae026d10e7a" Oct 30 19:03:37.968487 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:37.968448 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhqt" podUID="ee4d07b7-2acc-456e-932b-bfefb2f666df" Oct 30 19:03:38.111346 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:38.111307 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wk24l" event={"ID":"28f40fee-93d5-47de-89aa-943fa6b4efe7","Type":"ContainerStarted","Data":"8d1da7114ab5f055bbf3112be1562934e112889623037a804d65821aaaf80e3f"} Oct 30 19:03:38.113700 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:38.113672 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" event={"ID":"d3c525ed-5017-482f-ac5c-670002363472","Type":"ContainerStarted","Data":"4b063ca5aa09468a783c46e08bacdb2d783a8a93cc5b9f4353b7aac55ff1e04c"} Oct 30 19:03:38.142898 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:38.142853 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wk24l" podStartSLOduration=3.923404977 podStartE2EDuration="12.142837705s" totalImagesPullingTime="8.219432728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:38.142516999 +0000 UTC m=+12.730422176" watchObservedRunningTime="2026-10-30 19:03:38.142837705 +0000 UTC m=+12.730742885" Oct 30 19:03:38.178502 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:38.178458 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" podStartSLOduration=3.821544806 podStartE2EDuration="12.178444544s" totalImagesPullingTime="8.356899738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:38.17794935 +0000 UTC m=+12.765854526" watchObservedRunningTime="2026-10-30 19:03:38.178444544 +0000 UTC m=+12.766349721" Oct 30 19:03:39.119079 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:39.119032 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bdc45" event={"ID":"caef3ef2-f9af-4421-866e-61cc4febba1a","Type":"ContainerStarted","Data":"f684e0b2fdbef2052fe337a38ee170f5d157dbfc3323b3ce8d051d203effb9d9"} Oct 30 19:03:39.121769 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:39.121740 2574 generic.go:356] "Generic (PLEG): container finished" podID="7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad" containerID="ea2fbe1913283e26de0d0b3e0f5756cd711eff5aaaf13fed64c8e3fb58f9239e" exitCode=0 Oct 30 19:03:39.121919 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:39.121820 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k28t4" event={"ID":"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad","Type":"ContainerDied","Data":"ea2fbe1913283e26de0d0b3e0f5756cd711eff5aaaf13fed64c8e3fb58f9239e"} Oct 30 19:03:39.152650 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:39.152573 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bdc45" podStartSLOduration=2.821619591 podStartE2EDuration="13.152553786s" totalImagesPullingTime="10.330934195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:39.151872353 +0000 UTC m=+13.739777530" watchObservedRunningTime="2026-10-30 19:03:39.152553786 +0000 UTC m=+13.740458961" Oct 30 19:03:39.967915 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:39.967876 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:39.968119 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:39.967876 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:39.968119 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:39.968020 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx495" podUID="ad4cd78b-b491-4927-80ae-dae026d10e7a" Oct 30 19:03:39.968119 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:39.967893 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:39.968256 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:39.968134 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhqt" podUID="ee4d07b7-2acc-456e-932b-bfefb2f666df" Oct 30 19:03:39.968256 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:39.968231 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frj4v" podUID="93f0a70e-357e-4d65-b554-42363280bebd" Oct 30 19:03:40.128589 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:40.128562 2574 generic.go:356] "Generic (PLEG): container finished" podID="7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad" containerID="b265bf30eebb720ef475c3fc3343669c94f9f45efa12931a0c76c944e6fd493c" exitCode=0 Oct 30 19:03:40.129173 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:40.128690 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k28t4" event={"ID":"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad","Type":"ContainerDied","Data":"b265bf30eebb720ef475c3fc3343669c94f9f45efa12931a0c76c944e6fd493c"} Oct 30 19:03:41.699202 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:41.699171 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d26nk" Oct 30 19:03:41.699923 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:41.699903 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:03:41.700159 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:41.700121 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d26nk" Oct 30 19:03:41.967834 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:41.967549 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:41.967951 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:41.967553 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:41.967951 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:41.967913 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhqt" podUID="ee4d07b7-2acc-456e-932b-bfefb2f666df" Oct 30 19:03:41.967951 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:41.967562 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:41.968082 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:41.967956 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx495" podUID="ad4cd78b-b491-4927-80ae-dae026d10e7a" Oct 30 19:03:41.968082 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:41.968032 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frj4v" podUID="93f0a70e-357e-4d65-b554-42363280bebd" Oct 30 19:03:42.136156 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:42.136122 2574 generic.go:356] "Generic (PLEG): container finished" podID="7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad" containerID="d13be46182fc6c07d99a744067c1b668fd5bdcb1b4fad8bef35208985cde1066" exitCode=0 Oct 30 19:03:42.136344 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:42.136204 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k28t4" event={"ID":"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad","Type":"ContainerDied","Data":"d13be46182fc6c07d99a744067c1b668fd5bdcb1b4fad8bef35208985cde1066"} Oct 30 19:03:43.141194 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:43.140932 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" event={"ID":"f7537906-9d0e-4b31-9d76-45936780d55c","Type":"ContainerStarted","Data":"3b4b936c9b0cd9e3f9c544a1bc36fdad0179d460fa1445ee922912528d245b29"} Oct 30 19:03:43.141552 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:43.141236 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" event={"ID":"f7537906-9d0e-4b31-9d76-45936780d55c","Type":"ContainerStarted","Data":"0318ed77f5c7251fe15f108b0333614c61c4b08f68053cd8ce25340e20c6e3c3"} Oct 30 19:03:43.141552 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:43.141292 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" event={"ID":"f7537906-9d0e-4b31-9d76-45936780d55c","Type":"ContainerStarted","Data":"9fc878e68d9df58713194fcc08e71a65b4f9e996c22908564c7ec0718d59b513"} Oct 30 19:03:43.141552 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:43.141324 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" event={"ID":"f7537906-9d0e-4b31-9d76-45936780d55c","Type":"ContainerStarted","Data":"8e1c3a6ea22d0f881e7513538fc6b34e27dd336b4bef4f3fc7b33df40bc6ec68"} Oct 30 19:03:43.141552 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:43.141360 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" event={"ID":"f7537906-9d0e-4b31-9d76-45936780d55c","Type":"ContainerStarted","Data":"72551b74bdf3d2cd7ac47763cd0738621fb6caa0b5a7fa0f54a1931a89653af0"} Oct 30 19:03:43.144614 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:43.144587 2574 generic.go:356] "Generic (PLEG): container finished" podID="7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad" containerID="478056f779200b9270c7b20956e870efd2038f99ef4bb794f847106a12fbb09d" exitCode=0 Oct 30 19:03:43.144709 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:43.144643 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k28t4" event={"ID":"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad","Type":"ContainerDied","Data":"478056f779200b9270c7b20956e870efd2038f99ef4bb794f847106a12fbb09d"} Oct 30 19:03:43.682034 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:43.681990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:43.682253 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:43.682049 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrxn\" (UniqueName: \"kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn\") pod \"network-check-target-zwhqt\" (UID: \"ee4d07b7-2acc-456e-932b-bfefb2f666df\") " pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:43.682253 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:43.682134 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:43.682253 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:43.682197 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs podName:93f0a70e-357e-4d65-b554-42363280bebd nodeName:}" failed. No retries permitted until 2026-10-30 19:03:59.682179851 +0000 UTC m=+34.270085024 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs") pod "network-metrics-daemon-frj4v" (UID: "93f0a70e-357e-4d65-b554-42363280bebd") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:03:43.682253 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:43.682199 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:03:43.682253 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:43.682217 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:03:43.682253 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:43.682229 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ftrxn for pod openshift-network-diagnostics/network-check-target-zwhqt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:43.682457 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:43.682273 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn podName:ee4d07b7-2acc-456e-932b-bfefb2f666df nodeName:}" failed. No retries permitted until 2026-10-30 19:03:59.682257487 +0000 UTC m=+34.270162643 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ftrxn" (UniqueName: "kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn") pod "network-check-target-zwhqt" (UID: "ee4d07b7-2acc-456e-932b-bfefb2f666df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:03:43.782440 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:43.782413 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:43.782573 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:43.782556 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:43.782629 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:43.782617 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret podName:ad4cd78b-b491-4927-80ae-dae026d10e7a nodeName:}" failed. No retries permitted until 2026-10-30 19:03:59.782601653 +0000 UTC m=+34.370506814 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret") pod "global-pull-secret-syncer-hx495" (UID: "ad4cd78b-b491-4927-80ae-dae026d10e7a") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:03:43.968154 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:43.968063 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:43.968154 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:43.968093 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:43.968364 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:43.968155 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhqt" podUID="ee4d07b7-2acc-456e-932b-bfefb2f666df" Oct 30 19:03:43.968364 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:43.968212 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:43.968364 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:43.968293 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx495" podUID="ad4cd78b-b491-4927-80ae-dae026d10e7a" Oct 30 19:03:43.968514 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:43.968371 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frj4v" podUID="93f0a70e-357e-4d65-b554-42363280bebd" Oct 30 19:03:44.149950 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:44.149920 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" event={"ID":"f7537906-9d0e-4b31-9d76-45936780d55c","Type":"ContainerStarted","Data":"63661cca08e38f0ab0ef61f10f102b634d470ade415537819d473288d64d7f04"} Oct 30 19:03:44.153142 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:44.153117 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k28t4" event={"ID":"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad","Type":"ContainerStarted","Data":"fdb821cc98b68fc458d28f275c62fa77b108d7b9974dda926766fe291eb648ae"} Oct 30 19:03:44.210283 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:44.210226 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k28t4" podStartSLOduration=7.625934101 podStartE2EDuration="18.210213195s" totalImagesPullingTime="10.584279094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:44.200347726 +0000 UTC m=+18.788252905" watchObservedRunningTime="2026-10-30 19:03:44.210213195 +0000 UTC m=+18.798118373" Oct 30 19:03:45.967984 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:45.967951 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:45.968577 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:45.968061 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frj4v" podUID="93f0a70e-357e-4d65-b554-42363280bebd" Oct 30 19:03:45.968577 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:45.968145 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:45.968577 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:45.968253 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhqt" podUID="ee4d07b7-2acc-456e-932b-bfefb2f666df" Oct 30 19:03:45.968577 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:45.968303 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:45.968577 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:45.968383 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx495" podUID="ad4cd78b-b491-4927-80ae-dae026d10e7a" Oct 30 19:03:46.160603 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:46.160567 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" event={"ID":"f7537906-9d0e-4b31-9d76-45936780d55c","Type":"ContainerStarted","Data":"ff4205c0cb5ce2bfcdbfbac2e75f06acaa1ee0afea9194bfd5427cb960a7b336"} Oct 30 19:03:47.968179 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:47.968143 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:47.968872 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:47.968150 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:47.968872 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:47.968282 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:47.968872 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:47.968273 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhqt" podUID="ee4d07b7-2acc-456e-932b-bfefb2f666df" Oct 30 19:03:47.968872 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:47.968378 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frj4v" podUID="93f0a70e-357e-4d65-b554-42363280bebd" Oct 30 19:03:47.968872 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:47.968435 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx495" podUID="ad4cd78b-b491-4927-80ae-dae026d10e7a" Oct 30 19:03:48.168263 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:48.168228 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" event={"ID":"f7537906-9d0e-4b31-9d76-45936780d55c","Type":"ContainerStarted","Data":"6572fc717f144550a987066ac99033537f963ffe10faea4fb1059680463a8554"} Oct 30 19:03:48.168525 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:48.168489 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:48.168668 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:48.168561 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:48.183506 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:48.183484 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:48.236679 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:48.236593 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" podStartSLOduration=7.5336739470000005 podStartE2EDuration="22.236580222s" totalImagesPullingTime="14.702906275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:48.236346111 +0000 UTC m=+22.824251295" watchObservedRunningTime="2026-10-30 19:03:48.236580222 +0000 UTC m=+22.824485401" Oct 30 19:03:49.171272 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:49.171018 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:49.185628 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:49.185606 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:03:49.438982 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:49.438912 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zwhqt"] Oct 30 19:03:49.439138 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:49.439040 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:49.439204 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:49.439139 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhqt" podUID="ee4d07b7-2acc-456e-932b-bfefb2f666df" Oct 30 19:03:49.441704 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:49.441677 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hx495"] Oct 30 19:03:49.441822 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:49.441807 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:49.441944 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:49.441921 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx495" podUID="ad4cd78b-b491-4927-80ae-dae026d10e7a" Oct 30 19:03:49.442675 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:49.442650 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-frj4v"] Oct 30 19:03:49.442795 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:49.442744 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:49.442856 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:49.442827 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frj4v" podUID="93f0a70e-357e-4d65-b554-42363280bebd" Oct 30 19:03:50.968067 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:50.968031 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:50.968067 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:50.968055 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:50.968507 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:50.968139 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:50.968507 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:50.968146 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhqt" podUID="ee4d07b7-2acc-456e-932b-bfefb2f666df" Oct 30 19:03:50.968507 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:50.968212 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx495" podUID="ad4cd78b-b491-4927-80ae-dae026d10e7a" Oct 30 19:03:50.968507 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:50.968294 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frj4v" podUID="93f0a70e-357e-4d65-b554-42363280bebd" Oct 30 19:03:52.967384 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:52.967345 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:52.967384 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:52.967375 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:52.968112 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:52.967479 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-frj4v" podUID="93f0a70e-357e-4d65-b554-42363280bebd" Oct 30 19:03:52.968112 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:52.967502 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:52.968112 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:52.967565 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zwhqt" podUID="ee4d07b7-2acc-456e-932b-bfefb2f666df" Oct 30 19:03:52.968112 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:52.967629 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx495" podUID="ad4cd78b-b491-4927-80ae-dae026d10e7a" Oct 30 19:03:54.713032 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.712986 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeReady" Oct 30 19:03:54.713512 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.713154 2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 19:03:54.759808 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.759776 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d4d8bb96d-62jtz"] Oct 30 19:03:54.762563 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.762000 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sfbbt"] Oct 30 19:03:54.762563 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.762108 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.763595 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.763578 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pqg4r"] Oct 30 19:03:54.763709 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.763695 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-sfbbt" Oct 30 19:03:54.765372 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.765358 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-nxngq"] Oct 30 19:03:54.765496 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.765478 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pqg4r" Oct 30 19:03:54.767513 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.766871 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fs89c"] Oct 30 19:03:54.767513 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.767145 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" Oct 30 19:03:54.769113 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.768991 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fs89c" Oct 30 19:03:54.770250 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.770232 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5fd9cf966f-mn64l"] Oct 30 19:03:54.772763 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.772092 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k"] Oct 30 19:03:54.772763 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.772281 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.774136 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.774115 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s"] Oct 30 19:03:54.774243 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.774226 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:03:54.778982 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.778944 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 19:03:54.779130 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.778952 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 19:03:54.779130 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.778958 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 19:03:54.779301 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.779110 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l"] Oct 30 19:03:54.779301 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.779176 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" Oct 30 19:03:54.781330 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.781311 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wd8lm"] Oct 30 19:03:54.781425 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.781402 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" Oct 30 19:03:54.784279 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.784263 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-77d478f789-d787c"] Oct 30 19:03:54.784399 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.784382 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.786054 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.786034 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 19:03:54.786169 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.786151 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 19:03:54.786356 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.786339 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 19:03:54.786958 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.786938 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:03:54.788401 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.788381 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2"] Oct 30 19:03:54.788511 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.788495 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:54.791095 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.791079 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jdc67"] Oct 30 19:03:54.791222 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.791200 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:03:54.793607 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.793589 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6h7t5"] Oct 30 19:03:54.793875 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.793857 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jdc67" Oct 30 19:03:54.796369 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.796333 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cfdmh"] Oct 30 19:03:54.796637 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.796621 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:03:54.798425 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.798405 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 19:03:54.799519 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.799450 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jqrww"] Oct 30 19:03:54.800251 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.800233 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cfdmh" Oct 30 19:03:54.805956 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.804116 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ccd7w"] Oct 30 19:03:54.805956 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.805073 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jqrww" Oct 30 19:03:54.807928 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.807437 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sfbbt"] Oct 30 19:03:54.807928 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.807472 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d4d8bb96d-62jtz"] Oct 30 19:03:54.807928 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.807494 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pqg4r"] Oct 30 19:03:54.807928 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.807515 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fs89c"] Oct 30 19:03:54.807928 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.807523 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ccd7w" Oct 30 19:03:54.807928 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.807534 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k"] Oct 30 19:03:54.807928 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.807629 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jdc67"] Oct 30 19:03:54.807928 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.807652 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-nxngq"] Oct 30 19:03:54.807928 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.807674 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2"] Oct 30 19:03:54.807928 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.807725 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77d478f789-d787c"] Oct 30 19:03:54.807928 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.807746 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fd9cf966f-mn64l"] Oct 30 19:03:54.811792 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.811752 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wd8lm"] Oct 30 19:03:54.820271 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.820240 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6h7t5"] Oct 30 19:03:54.825715 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.825686 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cfdmh"] Oct 30 19:03:54.830921 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.830875 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l"] Oct 30 19:03:54.831094 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.830931 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ccd7w"] Oct 30 19:03:54.832160 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.832137 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s"] Oct 30 19:03:54.833670 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.833652 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jqrww"] Oct 30 19:03:54.860673 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.860649 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-installation-pull-secrets\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.860761 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.860677 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfkbg\" (UniqueName: \"kubernetes.io/projected/a8d10e22-9215-4fc2-8474-5bc68cb57f18-kube-api-access-bfkbg\") pod \"kas-connection-checker-685895c7c9-fs89c\" (UID: \"a8d10e22-9215-4fc2-8474-5bc68cb57f18\") " pod="kube-system/kas-connection-checker-685895c7c9-fs89c" Oct 30 19:03:54.860761 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.860696 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dfz6\" (UniqueName: \"kubernetes.io/projected/969393ae-3d0e-4f83-bf03-0396c386ffbf-kube-api-access-8dfz6\") pod \"volume-data-source-validator-58c9f5f46d-jdc67\" (UID: \"969393ae-3d0e-4f83-bf03-0396c386ffbf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jdc67" Oct 30 19:03:54.860761 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.860715 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nxngq\" (UID: \"bcf8833f-ed61-40f4-880c-2645ffae454b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" Oct 30 19:03:54.860933 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.860831 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-image-registry-private-configuration\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.860933 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.860911 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rxlq\" (UniqueName: \"kubernetes.io/projected/20403b1e-e479-45b9-bb85-52cf10725d86-kube-api-access-6rxlq\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:54.861046 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.860933 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-tmp\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.861046 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.860982 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ac3ab69-eead-4eca-af11-53fec376a6ac-ca-trust-extracted\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.861046 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861021 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-certificates\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.861208 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861089 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhzdr\" (UniqueName: \"kubernetes.io/projected/8ae35879-1912-43b0-a7a5-9ac064ce0dea-kube-api-access-zhzdr\") pod \"kas-connection-checker-685895c7c9-sfbbt\" (UID: \"8ae35879-1912-43b0-a7a5-9ac064ce0dea\") " pod="kube-system/kas-connection-checker-685895c7c9-sfbbt" Oct 30 19:03:54.861208 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861108 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-certificates\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.861208 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861127 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.861208 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861142 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:03:54.861208 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861161 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bphkr\" (UniqueName: \"kubernetes.io/projected/721cf657-76b1-4fdf-868d-d84cb9fa0b14-kube-api-access-bphkr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l\" (UID: \"721cf657-76b1-4fdf-868d-d84cb9fa0b14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" Oct 30 19:03:54.861442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861206 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd7cda5e-1a4b-45f8-9cab-f96111986c08-serving-cert\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:03:54.861442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861249 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-default-certificate\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:54.861442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861276 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-installation-pull-secrets\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.861442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861306 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-bound-sa-token\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.861442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861329 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/721cf657-76b1-4fdf-868d-d84cb9fa0b14-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l\" (UID: \"721cf657-76b1-4fdf-868d-d84cb9fa0b14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" Oct 30 19:03:54.861442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861358 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.861442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861392 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn7mf\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-kube-api-access-zn7mf\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.861442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861432 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x9z4\" (UniqueName: \"kubernetes.io/projected/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-kube-api-access-8x9z4\") pod \"service-ca-operator-544d749c8c-9ls9s\" (UID: \"c6188415-c8d5-4972-9d6d-f6e3bcf4b079\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" Oct 30 19:03:54.861660 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861479 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-bound-sa-token\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.861660 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861501 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvqk8\" (UniqueName: \"kubernetes.io/projected/57ee29d6-ad4d-4ed8-ae1a-877732a3737b-kube-api-access-tvqk8\") pod \"kas-connection-checker-685895c7c9-pqg4r\" (UID: \"57ee29d6-ad4d-4ed8-ae1a-877732a3737b\") " pod="kube-system/kas-connection-checker-685895c7c9-pqg4r" Oct 30 19:03:54.861660 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861519 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-config\") pod \"service-ca-operator-544d749c8c-9ls9s\" (UID: \"c6188415-c8d5-4972-9d6d-f6e3bcf4b079\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" Oct 30 19:03:54.861660 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861535 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd7cda5e-1a4b-45f8-9cab-f96111986c08-trusted-ca\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:03:54.861660 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861549 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6af08cb7-fa60-491b-88fd-07c1ace45779-ca-trust-extracted\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.861660 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861581 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:03:54.861660 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861608 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.861660 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861650 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-trusted-ca\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.861957 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861691 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bcf8833f-ed61-40f4-880c-2645ffae454b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nxngq\" (UID: \"bcf8833f-ed61-40f4-880c-2645ffae454b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" Oct 30 19:03:54.861957 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861743 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-stats-auth\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:54.861957 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd7cda5e-1a4b-45f8-9cab-f96111986c08-config\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:03:54.861957 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861825 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-snapshots\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.861957 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861917 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/721cf657-76b1-4fdf-868d-d84cb9fa0b14-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l\" (UID: \"721cf657-76b1-4fdf-868d-d84cb9fa0b14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" Oct 30 19:03:54.861957 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861946 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77qjb\" (UniqueName: \"kubernetes.io/projected/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-kube-api-access-77qjb\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.862266 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861967 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-trusted-ca\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.862266 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.861986 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-image-registry-private-configuration\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.862266 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.862021 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d65z\" (UniqueName: \"kubernetes.io/projected/b50175d2-e6bb-463a-9602-5bf9e4719a15-kube-api-access-2d65z\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:03:54.862266 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.862046 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dbws\" (UniqueName: \"kubernetes.io/projected/540944ec-348d-46e1-836b-9aeb07d16713-kube-api-access-6dbws\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:03:54.862266 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.862090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:54.862266 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.862115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp2w6\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-kube-api-access-sp2w6\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.862266 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.862130 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-service-ca-bundle\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.862266 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.862147 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9mcb\" (UniqueName: \"kubernetes.io/projected/bd7cda5e-1a4b-45f8-9cab-f96111986c08-kube-api-access-w9mcb\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:03:54.862266 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.862179 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-serving-cert\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.862266 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.862221 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:54.862266 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.862235 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/540944ec-348d-46e1-836b-9aeb07d16713-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:03:54.862266 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.862269 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:03:54.862831 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.862314 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-serving-cert\") pod \"service-ca-operator-544d749c8c-9ls9s\" (UID: \"c6188415-c8d5-4972-9d6d-f6e3bcf4b079\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" Oct 30 19:03:54.875214 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.875193 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 19:03:54.875443 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.875324 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:03:54.875443 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.875391 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 19:03:54.875443 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.875403 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 19:03:54.875443 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.875420 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 19:03:54.875443 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.875433 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 19:03:54.875657 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.875547 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8l2hk\"" Oct 30 19:03:54.875699 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.875656 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 19:03:54.875854 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.875843 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 19:03:54.875935 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.875888 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 19:03:54.875935 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.875901 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rgh44\"" Oct 30 19:03:54.876063 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.876032 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-c9nhn\"" Oct 30 19:03:54.876118 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.876092 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 19:03:54.876118 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.876100 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nft8r\"" Oct 30 19:03:54.876252 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.876121 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-8rqjm\"" Oct 30 19:03:54.876252 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.876133 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k9w5x\"" Oct 30 19:03:54.876252 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.876143 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wbp4l\"" Oct 30 19:03:54.876252 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.876121 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-hbw5j\"" Oct 30 19:03:54.876252 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.876151 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 19:03:54.876252 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.876161 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:03:54.876252 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.876133 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-t9l5d\"" Oct 30 19:03:54.876252 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.876177 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rzhnr\"" Oct 30 19:03:54.876252 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.876198 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-z6lkw\"" Oct 30 19:03:54.876252 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.876204 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bwrqq\"" Oct 30 19:03:54.876252 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.876224 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ghkqg\"" Oct 30 19:03:54.884940 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.884920 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:03:54.885037 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.884962 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:03:54.885097 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885083 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:03:54.885136 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885128 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 19:03:54.885185 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885161 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 19:03:54.885185 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885166 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:03:54.885293 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885192 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:03:54.885293 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885198 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:03:54.885293 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885238 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 19:03:54.885293 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885277 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:03:54.885495 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885308 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 19:03:54.885495 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885358 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:03:54.885495 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885419 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 19:03:54.885495 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885482 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:03:54.886125 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885827 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 19:03:54.886125 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885842 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:03:54.886125 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885858 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:03:54.886125 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885885 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 19:03:54.886125 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885887 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:03:54.886125 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.885931 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 19:03:54.886125 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.886093 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:03:54.886468 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.886199 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:03:54.886468 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.886215 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:03:54.886468 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.886275 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 19:03:54.886468 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.886371 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 19:03:54.886468 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.886454 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 19:03:54.886710 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.886634 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 19:03:54.897000 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.896980 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 19:03:54.897102 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.897057 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 19:03:54.963559 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.963480 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dbws\" (UniqueName: \"kubernetes.io/projected/540944ec-348d-46e1-836b-9aeb07d16713-kube-api-access-6dbws\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:03:54.963559 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.963515 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:54.963559 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.963532 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp2w6\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-kube-api-access-sp2w6\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.963559 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.963555 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-service-ca-bundle\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.963890 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.963582 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9mcb\" (UniqueName: \"kubernetes.io/projected/bd7cda5e-1a4b-45f8-9cab-f96111986c08-kube-api-access-w9mcb\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:03:54.963890 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.963606 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-serving-cert\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.963890 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:54.963684 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle podName:20403b1e-e479-45b9-bb85-52cf10725d86 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:55.463662722 +0000 UTC m=+30.051567898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle") pod "router-default-77d478f789-d787c" (UID: "20403b1e-e479-45b9-bb85-52cf10725d86") : configmap references non-existent config key: service-ca.crt Oct 30 19:03:54.963890 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.963730 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e626b03-28b0-4840-bbe2-518834311815-cert\") pod \"ingress-canary-ccd7w\" (UID: \"3e626b03-28b0-4840-bbe2-518834311815\") " pod="openshift-ingress-canary/ingress-canary-ccd7w" Oct 30 19:03:54.963890 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.963785 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:54.963890 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.963833 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/540944ec-348d-46e1-836b-9aeb07d16713-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:03:54.963890 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.963862 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:03:54.963890 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.963904 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-serving-cert\") pod \"service-ca-operator-544d749c8c-9ls9s\" (UID: \"c6188415-c8d5-4972-9d6d-f6e3bcf4b079\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" Oct 30 19:03:54.963890 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:54.963907 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:03:54.964326 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:54.963969 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:03:54.964326 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:54.963972 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs podName:20403b1e-e479-45b9-bb85-52cf10725d86 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:55.463957203 +0000 UTC m=+30.051862374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs") pod "router-default-77d478f789-d787c" (UID: "20403b1e-e479-45b9-bb85-52cf10725d86") : secret "router-metrics-certs-default" not found Oct 30 19:03:54.964326 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.964080 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-installation-pull-secrets\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.964326 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:54.964104 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls podName:540944ec-348d-46e1-836b-9aeb07d16713 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:55.464093349 +0000 UTC m=+30.051998506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x4r6k" (UID: "540944ec-348d-46e1-836b-9aeb07d16713") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:03:54.964326 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.964135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfkbg\" (UniqueName: \"kubernetes.io/projected/a8d10e22-9215-4fc2-8474-5bc68cb57f18-kube-api-access-bfkbg\") pod \"kas-connection-checker-685895c7c9-fs89c\" (UID: \"a8d10e22-9215-4fc2-8474-5bc68cb57f18\") " pod="kube-system/kas-connection-checker-685895c7c9-fs89c" Oct 30 19:03:54.964326 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.964171 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kd76\" (UniqueName: \"kubernetes.io/projected/bfcde492-9464-42cc-a01c-84d9eec4135a-kube-api-access-9kd76\") pod \"network-check-source-6f78d59f99-cfdmh\" (UID: \"bfcde492-9464-42cc-a01c-84d9eec4135a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cfdmh" Oct 30 19:03:54.964326 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.964223 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dfz6\" (UniqueName: \"kubernetes.io/projected/969393ae-3d0e-4f83-bf03-0396c386ffbf-kube-api-access-8dfz6\") pod \"volume-data-source-validator-58c9f5f46d-jdc67\" (UID: \"969393ae-3d0e-4f83-bf03-0396c386ffbf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jdc67" Oct 30 19:03:54.964326 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.964249 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nxngq\" (UID: \"bcf8833f-ed61-40f4-880c-2645ffae454b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" Oct 30 19:03:54.964326 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.964275 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdw7x\" (UniqueName: \"kubernetes.io/projected/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-kube-api-access-bdw7x\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:03:54.964326 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.964311 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-service-ca-bundle\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.964882 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.964340 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-image-registry-private-configuration\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.964882 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:54.964419 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:03:54.964882 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:54.964463 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert podName:bcf8833f-ed61-40f4-880c-2645ffae454b nodeName:}" failed. No retries permitted until 2026-10-30 19:03:55.464448797 +0000 UTC m=+30.052353979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nxngq" (UID: "bcf8833f-ed61-40f4-880c-2645ffae454b") : secret "networking-console-plugin-cert" not found Oct 30 19:03:54.964882 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.964512 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rxlq\" (UniqueName: \"kubernetes.io/projected/20403b1e-e479-45b9-bb85-52cf10725d86-kube-api-access-6rxlq\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:54.964882 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.964535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-tmp\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.964882 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.964567 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtd7p\" (UniqueName: \"kubernetes.io/projected/3e626b03-28b0-4840-bbe2-518834311815-kube-api-access-wtd7p\") pod \"ingress-canary-ccd7w\" (UID: \"3e626b03-28b0-4840-bbe2-518834311815\") " pod="openshift-ingress-canary/ingress-canary-ccd7w" Oct 30 19:03:54.964882 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.964616 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ac3ab69-eead-4eca-af11-53fec376a6ac-ca-trust-extracted\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.964882 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.964641 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-certificates\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.964882 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.964657 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/540944ec-348d-46e1-836b-9aeb07d16713-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:03:54.964882 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.964666 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhzdr\" (UniqueName: \"kubernetes.io/projected/8ae35879-1912-43b0-a7a5-9ac064ce0dea-kube-api-access-zhzdr\") pod \"kas-connection-checker-685895c7c9-sfbbt\" (UID: \"8ae35879-1912-43b0-a7a5-9ac064ce0dea\") " pod="kube-system/kas-connection-checker-685895c7c9-sfbbt" Oct 30 19:03:54.964882 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.964691 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-certificates\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.965446 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.964988 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-tmp\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.965446 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.965092 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ac3ab69-eead-4eca-af11-53fec376a6ac-ca-trust-extracted\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.965446 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.965326 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.965446 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:54.965414 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:03:54.965446 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:54.965426 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4d8bb96d-62jtz: secret "image-registry-tls" not found Oct 30 19:03:54.965694 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:54.965463 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls podName:6af08cb7-fa60-491b-88fd-07c1ace45779 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:55.465450113 +0000 UTC m=+30.053355283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls") pod "image-registry-5d4d8bb96d-62jtz" (UID: "6af08cb7-fa60-491b-88fd-07c1ace45779") : secret "image-registry-tls" not found Oct 30 19:03:54.965694 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.965485 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-certificates\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.965694 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.965495 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:03:54.965694 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:54.965552 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:03:54.965694 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.965544 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bphkr\" (UniqueName: \"kubernetes.io/projected/721cf657-76b1-4fdf-868d-d84cb9fa0b14-kube-api-access-bphkr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l\" (UID: \"721cf657-76b1-4fdf-868d-d84cb9fa0b14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" Oct 30 19:03:54.965694 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.965546 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-certificates\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.965694 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:54.965589 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls podName:b50175d2-e6bb-463a-9602-5bf9e4719a15 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:55.465578273 +0000 UTC m=+30.053483445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lx6v2" (UID: "b50175d2-e6bb-463a-9602-5bf9e4719a15") : secret "samples-operator-tls" not found Oct 30 19:03:54.965694 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.965627 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd7cda5e-1a4b-45f8-9cab-f96111986c08-serving-cert\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:03:54.965694 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.965689 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-default-certificate\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:54.966170 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.965715 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-installation-pull-secrets\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.966170 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.965767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-bound-sa-token\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.966170 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.965794 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/721cf657-76b1-4fdf-868d-d84cb9fa0b14-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l\" (UID: \"721cf657-76b1-4fdf-868d-d84cb9fa0b14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" Oct 30 19:03:54.967000 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.965822 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.967000 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.966379 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn7mf\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-kube-api-access-zn7mf\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.967000 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.966477 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x9z4\" (UniqueName: \"kubernetes.io/projected/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-kube-api-access-8x9z4\") pod \"service-ca-operator-544d749c8c-9ls9s\" (UID: \"c6188415-c8d5-4972-9d6d-f6e3bcf4b079\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" Oct 30 19:03:54.967000 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.966509 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-bound-sa-token\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.967000 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.966553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvqk8\" (UniqueName: \"kubernetes.io/projected/57ee29d6-ad4d-4ed8-ae1a-877732a3737b-kube-api-access-tvqk8\") pod \"kas-connection-checker-685895c7c9-pqg4r\" (UID: \"57ee29d6-ad4d-4ed8-ae1a-877732a3737b\") " pod="kube-system/kas-connection-checker-685895c7c9-pqg4r" Oct 30 19:03:54.967000 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.966564 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/721cf657-76b1-4fdf-868d-d84cb9fa0b14-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l\" (UID: \"721cf657-76b1-4fdf-868d-d84cb9fa0b14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" Oct 30 19:03:54.967000 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.966587 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-config\") pod \"service-ca-operator-544d749c8c-9ls9s\" (UID: \"c6188415-c8d5-4972-9d6d-f6e3bcf4b079\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" Oct 30 19:03:54.967000 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.966613 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd7cda5e-1a4b-45f8-9cab-f96111986c08-trusted-ca\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:03:54.967000 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.966650 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6af08cb7-fa60-491b-88fd-07c1ace45779-ca-trust-extracted\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.967000 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.966675 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:03:54.967000 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.966699 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.967000 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.966727 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-trusted-ca\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.967000 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.966759 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bcf8833f-ed61-40f4-880c-2645ffae454b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nxngq\" (UID: \"bcf8833f-ed61-40f4-880c-2645ffae454b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" Oct 30 19:03:54.967000 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.966819 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:03:54.967000 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:54.966850 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:03:54.967000 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:54.966863 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd9cf966f-mn64l: secret "image-registry-tls" not found Oct 30 19:03:54.967794 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:54.966906 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls podName:3ac3ab69-eead-4eca-af11-53fec376a6ac nodeName:}" failed. No retries permitted until 2026-10-30 19:03:55.466892399 +0000 UTC m=+30.054797577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls") pod "image-registry-5fd9cf966f-mn64l" (UID: "3ac3ab69-eead-4eca-af11-53fec376a6ac") : secret "image-registry-tls" not found Oct 30 19:03:54.968907 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.968882 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-config\") pod \"service-ca-operator-544d749c8c-9ls9s\" (UID: \"c6188415-c8d5-4972-9d6d-f6e3bcf4b079\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" Oct 30 19:03:54.969562 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.969514 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd7cda5e-1a4b-45f8-9cab-f96111986c08-trusted-ca\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:03:54.969727 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.969704 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:03:54.969970 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.969943 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bcf8833f-ed61-40f4-880c-2645ffae454b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nxngq\" (UID: \"bcf8833f-ed61-40f4-880c-2645ffae454b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" Oct 30 19:03:54.970324 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.970299 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6af08cb7-fa60-491b-88fd-07c1ace45779-ca-trust-extracted\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.970578 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.970546 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-serving-cert\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.970814 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.970780 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-installation-pull-secrets\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.970814 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.970781 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd7cda5e-1a4b-45f8-9cab-f96111986c08-serving-cert\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:03:54.971019 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.970833 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-default-certificate\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:54.971019 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.966851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-stats-auth\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:54.971019 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.970912 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd7cda5e-1a4b-45f8-9cab-f96111986c08-config\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:03:54.971019 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.970949 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-snapshots\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.971019 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.970993 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/721cf657-76b1-4fdf-868d-d84cb9fa0b14-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l\" (UID: \"721cf657-76b1-4fdf-868d-d84cb9fa0b14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" Oct 30 19:03:54.971291 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.971044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77qjb\" (UniqueName: \"kubernetes.io/projected/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-kube-api-access-77qjb\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.971291 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.971085 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-trusted-ca\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.971291 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.971121 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-config-volume\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:03:54.971291 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.971149 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-tmp-dir\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:03:54.971291 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.971188 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-image-registry-private-configuration\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.971291 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.971217 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-stats-auth\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:54.971291 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.971282 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-image-registry-private-configuration\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.971570 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.971286 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-trusted-ca\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.971570 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.971531 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.971570 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.971221 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d65z\" (UniqueName: \"kubernetes.io/projected/b50175d2-e6bb-463a-9602-5bf9e4719a15-kube-api-access-2d65z\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:03:54.971717 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.971654 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-snapshots\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.971859 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.971828 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-serving-cert\") pod \"service-ca-operator-544d749c8c-9ls9s\" (UID: \"c6188415-c8d5-4972-9d6d-f6e3bcf4b079\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" Oct 30 19:03:54.972259 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.972238 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd7cda5e-1a4b-45f8-9cab-f96111986c08-config\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:03:54.972493 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.972466 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:54.973632 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.972680 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:54.973632 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.972874 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-trusted-ca\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.973632 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.972641 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:54.973632 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.973164 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-installation-pull-secrets\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.973883 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.973823 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/721cf657-76b1-4fdf-868d-d84cb9fa0b14-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l\" (UID: \"721cf657-76b1-4fdf-868d-d84cb9fa0b14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" Oct 30 19:03:54.974222 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.974201 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-image-registry-private-configuration\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.982486 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.982364 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfkbg\" (UniqueName: \"kubernetes.io/projected/a8d10e22-9215-4fc2-8474-5bc68cb57f18-kube-api-access-bfkbg\") pod \"kas-connection-checker-685895c7c9-fs89c\" (UID: \"a8d10e22-9215-4fc2-8474-5bc68cb57f18\") " pod="kube-system/kas-connection-checker-685895c7c9-fs89c" Oct 30 19:03:54.982586 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.982568 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9mcb\" (UniqueName: \"kubernetes.io/projected/bd7cda5e-1a4b-45f8-9cab-f96111986c08-kube-api-access-w9mcb\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:03:54.982766 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.982745 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dbws\" (UniqueName: \"kubernetes.io/projected/540944ec-348d-46e1-836b-9aeb07d16713-kube-api-access-6dbws\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:03:54.983276 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.983251 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rxlq\" (UniqueName: \"kubernetes.io/projected/20403b1e-e479-45b9-bb85-52cf10725d86-kube-api-access-6rxlq\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:54.983514 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.983499 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhzdr\" (UniqueName: \"kubernetes.io/projected/8ae35879-1912-43b0-a7a5-9ac064ce0dea-kube-api-access-zhzdr\") pod \"kas-connection-checker-685895c7c9-sfbbt\" (UID: \"8ae35879-1912-43b0-a7a5-9ac064ce0dea\") " pod="kube-system/kas-connection-checker-685895c7c9-sfbbt" Oct 30 19:03:54.984381 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.984353 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dfz6\" (UniqueName: \"kubernetes.io/projected/969393ae-3d0e-4f83-bf03-0396c386ffbf-kube-api-access-8dfz6\") pod \"volume-data-source-validator-58c9f5f46d-jdc67\" (UID: \"969393ae-3d0e-4f83-bf03-0396c386ffbf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jdc67" Oct 30 19:03:54.984381 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.984370 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp2w6\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-kube-api-access-sp2w6\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.985469 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.985449 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-s89t7\"" Oct 30 19:03:54.985585 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.985490 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2sn6r\"" Oct 30 19:03:54.985585 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.985497 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:03:54.985585 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.985590 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:03:54.985764 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.985612 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4bfbh\"" Oct 30 19:03:54.989158 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.989133 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-bound-sa-token\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:54.989511 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.989456 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x9z4\" (UniqueName: \"kubernetes.io/projected/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-kube-api-access-8x9z4\") pod \"service-ca-operator-544d749c8c-9ls9s\" (UID: \"c6188415-c8d5-4972-9d6d-f6e3bcf4b079\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" Oct 30 19:03:54.989597 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.989524 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bphkr\" (UniqueName: \"kubernetes.io/projected/721cf657-76b1-4fdf-868d-d84cb9fa0b14-kube-api-access-bphkr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l\" (UID: \"721cf657-76b1-4fdf-868d-d84cb9fa0b14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" Oct 30 19:03:54.989796 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.989775 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-bound-sa-token\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:54.989954 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.989934 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d65z\" (UniqueName: \"kubernetes.io/projected/b50175d2-e6bb-463a-9602-5bf9e4719a15-kube-api-access-2d65z\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:03:54.990126 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.990111 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvqk8\" (UniqueName: \"kubernetes.io/projected/57ee29d6-ad4d-4ed8-ae1a-877732a3737b-kube-api-access-tvqk8\") pod \"kas-connection-checker-685895c7c9-pqg4r\" (UID: \"57ee29d6-ad4d-4ed8-ae1a-877732a3737b\") " pod="kube-system/kas-connection-checker-685895c7c9-pqg4r" Oct 30 19:03:54.990461 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.990439 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77qjb\" (UniqueName: \"kubernetes.io/projected/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-kube-api-access-77qjb\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:54.991155 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:54.991139 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn7mf\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-kube-api-access-zn7mf\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:55.071976 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.071936 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kd76\" (UniqueName: \"kubernetes.io/projected/bfcde492-9464-42cc-a01c-84d9eec4135a-kube-api-access-9kd76\") pod \"network-check-source-6f78d59f99-cfdmh\" (UID: \"bfcde492-9464-42cc-a01c-84d9eec4135a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cfdmh" Oct 30 19:03:55.071976 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.071986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdw7x\" (UniqueName: \"kubernetes.io/projected/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-kube-api-access-bdw7x\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:03:55.072254 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.072032 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtd7p\" (UniqueName: \"kubernetes.io/projected/3e626b03-28b0-4840-bbe2-518834311815-kube-api-access-wtd7p\") pod \"ingress-canary-ccd7w\" (UID: \"3e626b03-28b0-4840-bbe2-518834311815\") " pod="openshift-ingress-canary/ingress-canary-ccd7w" Oct 30 19:03:55.072254 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.072110 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:03:55.072254 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.072139 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-config-volume\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:03:55.072254 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.072153 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-tmp-dir\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:03:55.072254 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.072195 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e626b03-28b0-4840-bbe2-518834311815-cert\") pod \"ingress-canary-ccd7w\" (UID: \"3e626b03-28b0-4840-bbe2-518834311815\") " pod="openshift-ingress-canary/ingress-canary-ccd7w" Oct 30 19:03:55.072469 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.072254 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:03:55.072469 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.072316 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls podName:a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b nodeName:}" failed. No retries permitted until 2026-10-30 19:03:55.572297793 +0000 UTC m=+30.160202960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls") pod "dns-default-jqrww" (UID: "a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b") : secret "dns-default-metrics-tls" not found Oct 30 19:03:55.072574 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.072557 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-tmp-dir\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:03:55.072711 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.072692 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-config-volume\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:03:55.074465 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.074449 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e626b03-28b0-4840-bbe2-518834311815-cert\") pod \"ingress-canary-ccd7w\" (UID: \"3e626b03-28b0-4840-bbe2-518834311815\") " pod="openshift-ingress-canary/ingress-canary-ccd7w" Oct 30 19:03:55.085685 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.085659 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-sfbbt" Oct 30 19:03:55.090253 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.090228 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtd7p\" (UniqueName: \"kubernetes.io/projected/3e626b03-28b0-4840-bbe2-518834311815-kube-api-access-wtd7p\") pod \"ingress-canary-ccd7w\" (UID: \"3e626b03-28b0-4840-bbe2-518834311815\") " pod="openshift-ingress-canary/ingress-canary-ccd7w" Oct 30 19:03:55.090661 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.090642 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kd76\" (UniqueName: \"kubernetes.io/projected/bfcde492-9464-42cc-a01c-84d9eec4135a-kube-api-access-9kd76\") pod \"network-check-source-6f78d59f99-cfdmh\" (UID: \"bfcde492-9464-42cc-a01c-84d9eec4135a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cfdmh" Oct 30 19:03:55.090744 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.090642 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdw7x\" (UniqueName: \"kubernetes.io/projected/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-kube-api-access-bdw7x\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:03:55.094503 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.094488 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pqg4r" Oct 30 19:03:55.110953 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.110928 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fs89c" Oct 30 19:03:55.139918 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.139886 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" Oct 30 19:03:55.146665 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.146030 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" Oct 30 19:03:55.152703 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.152471 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:03:55.171091 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.170617 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jdc67" Oct 30 19:03:55.184416 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.184129 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:03:55.186262 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.184592 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cfdmh" Oct 30 19:03:55.194874 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.194684 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ccd7w" Oct 30 19:03:55.391423 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.391394 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pqg4r"] Oct 30 19:03:55.392851 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.392828 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sfbbt"] Oct 30 19:03:55.393179 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:55.393134 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae35879_1912_43b0_a7a5_9ac064ce0dea.slice/crio-a5e0b634a45229b1af26839f8414ee1ac9f2c372dd0acb5e8862e567bf6889d4 WatchSource:0}: Error finding container a5e0b634a45229b1af26839f8414ee1ac9f2c372dd0acb5e8862e567bf6889d4: Status 404 returned error can't find the container with id a5e0b634a45229b1af26839f8414ee1ac9f2c372dd0acb5e8862e567bf6889d4 Oct 30 19:03:55.393915 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:55.393891 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ee29d6_ad4d_4ed8_ae1a_877732a3737b.slice/crio-70a7967543e8a1c5cd506bff801efa08d3a298510b88e1a03def33cee2a138d1 WatchSource:0}: Error finding container 70a7967543e8a1c5cd506bff801efa08d3a298510b88e1a03def33cee2a138d1: Status 404 returned error can't find the container with id 70a7967543e8a1c5cd506bff801efa08d3a298510b88e1a03def33cee2a138d1 Oct 30 19:03:55.409714 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.409688 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fs89c"] Oct 30 19:03:55.412775 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:55.412748 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d10e22_9215_4fc2_8474_5bc68cb57f18.slice/crio-e9365e876ff21c8086855f6e1563d090b4aebb43c65f350da97ddaf74a361471 WatchSource:0}: Error finding container e9365e876ff21c8086855f6e1563d090b4aebb43c65f350da97ddaf74a361471: Status 404 returned error can't find the container with id e9365e876ff21c8086855f6e1563d090b4aebb43c65f350da97ddaf74a361471 Oct 30 19:03:55.454272 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.454196 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s"] Oct 30 19:03:55.460067 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:55.460033 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6188415_c8d5_4972_9d6d_f6e3bcf4b079.slice/crio-f641733acf22c32e20702d19d3e0226b46203b223b7985d7ca60947e7541184f WatchSource:0}: Error finding container f641733acf22c32e20702d19d3e0226b46203b223b7985d7ca60947e7541184f: Status 404 returned error can't find the container with id f641733acf22c32e20702d19d3e0226b46203b223b7985d7ca60947e7541184f Oct 30 19:03:55.496858 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.486292 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l"] Oct 30 19:03:55.496858 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.488235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:55.496858 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.488282 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:55.496858 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.488321 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:03:55.496858 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.488402 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nxngq\" (UID: \"bcf8833f-ed61-40f4-880c-2645ffae454b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" Oct 30 19:03:55.496858 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.488456 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:55.496858 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.488490 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:03:55.496858 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.488558 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:55.496858 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.488703 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:03:55.496858 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.488716 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd9cf966f-mn64l: secret "image-registry-tls" not found Oct 30 19:03:55.496858 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.488778 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls podName:3ac3ab69-eead-4eca-af11-53fec376a6ac nodeName:}" failed. No retries permitted until 2026-10-30 19:03:56.488754883 +0000 UTC m=+31.076660040 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls") pod "image-registry-5fd9cf966f-mn64l" (UID: "3ac3ab69-eead-4eca-af11-53fec376a6ac") : secret "image-registry-tls" not found Oct 30 19:03:55.496858 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.488880 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle podName:20403b1e-e479-45b9-bb85-52cf10725d86 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:56.488870972 +0000 UTC m=+31.076776129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle") pod "router-default-77d478f789-d787c" (UID: "20403b1e-e479-45b9-bb85-52cf10725d86") : configmap references non-existent config key: service-ca.crt Oct 30 19:03:55.496858 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.488949 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:03:55.496858 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.488987 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs podName:20403b1e-e479-45b9-bb85-52cf10725d86 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:56.488976905 +0000 UTC m=+31.076882075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs") pod "router-default-77d478f789-d787c" (UID: "20403b1e-e479-45b9-bb85-52cf10725d86") : secret "router-metrics-certs-default" not found Oct 30 19:03:55.496858 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.489087 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:03:55.496858 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.489135 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls podName:540944ec-348d-46e1-836b-9aeb07d16713 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:56.489122055 +0000 UTC m=+31.077027213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x4r6k" (UID: "540944ec-348d-46e1-836b-9aeb07d16713") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:03:55.498301 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.489204 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:03:55.498301 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.489248 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert podName:bcf8833f-ed61-40f4-880c-2645ffae454b nodeName:}" failed. No retries permitted until 2026-10-30 19:03:56.489226056 +0000 UTC m=+31.077131211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nxngq" (UID: "bcf8833f-ed61-40f4-880c-2645ffae454b") : secret "networking-console-plugin-cert" not found Oct 30 19:03:55.498301 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.489322 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:03:55.498301 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.489333 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4d8bb96d-62jtz: secret "image-registry-tls" not found Oct 30 19:03:55.498301 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.489373 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls podName:6af08cb7-fa60-491b-88fd-07c1ace45779 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:56.489360604 +0000 UTC m=+31.077265762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls") pod "image-registry-5d4d8bb96d-62jtz" (UID: "6af08cb7-fa60-491b-88fd-07c1ace45779") : secret "image-registry-tls" not found Oct 30 19:03:55.498301 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.489448 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:03:55.498301 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.489490 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls podName:b50175d2-e6bb-463a-9602-5bf9e4719a15 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:56.489480879 +0000 UTC m=+31.077386037 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lx6v2" (UID: "b50175d2-e6bb-463a-9602-5bf9e4719a15") : secret "samples-operator-tls" not found Oct 30 19:03:55.519738 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.519715 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wd8lm"] Oct 30 19:03:55.522364 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:55.522340 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d31a53_ae1b_4d46_aa8c_6c00e89ed539.slice/crio-74050fbb7ca681a4b0d0176a5d938d780a131034dfe048fef03ee3dcbc83aded WatchSource:0}: Error finding container 74050fbb7ca681a4b0d0176a5d938d780a131034dfe048fef03ee3dcbc83aded: Status 404 returned error can't find the container with id 74050fbb7ca681a4b0d0176a5d938d780a131034dfe048fef03ee3dcbc83aded Oct 30 19:03:55.525445 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.525424 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6h7t5"] Oct 30 19:03:55.531085 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:55.531063 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7cda5e_1a4b_45f8_9cab_f96111986c08.slice/crio-1107f3e4351949dcfe0a9516bdcf58cd346932add8fa00e07c869b68efcea9f1 WatchSource:0}: Error finding container 1107f3e4351949dcfe0a9516bdcf58cd346932add8fa00e07c869b68efcea9f1: Status 404 returned error can't find the container with id 1107f3e4351949dcfe0a9516bdcf58cd346932add8fa00e07c869b68efcea9f1 Oct 30 19:03:55.545046 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.544993 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cfdmh"] Oct 30 19:03:55.547451 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:55.547428 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfcde492_9464_42cc_a01c_84d9eec4135a.slice/crio-70748cb186264e6b7d4b61ec14812cd80f483a9b29732084126e01652d7db72e WatchSource:0}: Error finding container 70748cb186264e6b7d4b61ec14812cd80f483a9b29732084126e01652d7db72e: Status 404 returned error can't find the container with id 70748cb186264e6b7d4b61ec14812cd80f483a9b29732084126e01652d7db72e Oct 30 19:03:55.566882 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.566461 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jdc67"] Oct 30 19:03:55.567882 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.567855 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ccd7w"] Oct 30 19:03:55.568865 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:55.568839 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e626b03_28b0_4840_bbe2_518834311815.slice/crio-bb8e5379f32304111cf0192bb8be7c0f2484ec432c185480ca4af01bb1f32133 WatchSource:0}: Error finding container bb8e5379f32304111cf0192bb8be7c0f2484ec432c185480ca4af01bb1f32133: Status 404 returned error can't find the container with id bb8e5379f32304111cf0192bb8be7c0f2484ec432c185480ca4af01bb1f32133 Oct 30 19:03:55.571728 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:03:55.570828 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969393ae_3d0e_4f83_bf03_0396c386ffbf.slice/crio-376eabfad4be616716b578f3918e6800b8fe2eb6a7ce7b84cb32e95810229ef3 WatchSource:0}: Error finding container 376eabfad4be616716b578f3918e6800b8fe2eb6a7ce7b84cb32e95810229ef3: Status 404 returned error can't find the container with id 376eabfad4be616716b578f3918e6800b8fe2eb6a7ce7b84cb32e95810229ef3 Oct 30 19:03:55.589772 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:55.589749 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:03:55.589935 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.589918 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:03:55.589997 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:55.589984 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls podName:a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b nodeName:}" failed. No retries permitted until 2026-10-30 19:03:56.589963782 +0000 UTC m=+31.177868957 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls") pod "dns-default-jqrww" (UID: "a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b") : secret "dns-default-metrics-tls" not found Oct 30 19:03:56.200343 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.200274 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" event={"ID":"721cf657-76b1-4fdf-868d-d84cb9fa0b14","Type":"ContainerStarted","Data":"aaa13a46135b4def3571bba4900cdd96c05d508a0aea8d03d55b21e79f216836"} Oct 30 19:03:56.202599 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.202570 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" event={"ID":"c6188415-c8d5-4972-9d6d-f6e3bcf4b079","Type":"ContainerStarted","Data":"f641733acf22c32e20702d19d3e0226b46203b223b7985d7ca60947e7541184f"} Oct 30 19:03:56.205753 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.204811 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fs89c" event={"ID":"a8d10e22-9215-4fc2-8474-5bc68cb57f18","Type":"ContainerStarted","Data":"5b7239df824bb30b823fa27e654ceea6c83bf5ec8782448b230b8fdbd79148a3"} Oct 30 19:03:56.205753 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.204860 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fs89c" event={"ID":"a8d10e22-9215-4fc2-8474-5bc68cb57f18","Type":"ContainerStarted","Data":"e9365e876ff21c8086855f6e1563d090b4aebb43c65f350da97ddaf74a361471"} Oct 30 19:03:56.210183 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.209635 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pqg4r" event={"ID":"57ee29d6-ad4d-4ed8-ae1a-877732a3737b","Type":"ContainerStarted","Data":"e863f9c297e1015e709191434e626a606648ec01a31ce2ca14ce41a51a2cfee8"} Oct 30 19:03:56.210183 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.209683 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pqg4r" event={"ID":"57ee29d6-ad4d-4ed8-ae1a-877732a3737b","Type":"ContainerStarted","Data":"70a7967543e8a1c5cd506bff801efa08d3a298510b88e1a03def33cee2a138d1"} Oct 30 19:03:56.212160 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.212130 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ccd7w" event={"ID":"3e626b03-28b0-4840-bbe2-518834311815","Type":"ContainerStarted","Data":"bb8e5379f32304111cf0192bb8be7c0f2484ec432c185480ca4af01bb1f32133"} Oct 30 19:03:56.220732 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.220699 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wd8lm" event={"ID":"11d31a53-ae1b-4d46-aa8c-6c00e89ed539","Type":"ContainerStarted","Data":"74050fbb7ca681a4b0d0176a5d938d780a131034dfe048fef03ee3dcbc83aded"} Oct 30 19:03:56.223733 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.223189 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-sfbbt" event={"ID":"8ae35879-1912-43b0-a7a5-9ac064ce0dea","Type":"ContainerStarted","Data":"9e0c5d8292a14c57052ea8bdbdd176d0741721bf946ecaaa5249efef21675d2e"} Oct 30 19:03:56.223733 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.223235 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-sfbbt" event={"ID":"8ae35879-1912-43b0-a7a5-9ac064ce0dea","Type":"ContainerStarted","Data":"a5e0b634a45229b1af26839f8414ee1ac9f2c372dd0acb5e8862e567bf6889d4"} Oct 30 19:03:56.225418 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.225361 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cfdmh" event={"ID":"bfcde492-9464-42cc-a01c-84d9eec4135a","Type":"ContainerStarted","Data":"70748cb186264e6b7d4b61ec14812cd80f483a9b29732084126e01652d7db72e"} Oct 30 19:03:56.226985 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.226956 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jdc67" event={"ID":"969393ae-3d0e-4f83-bf03-0396c386ffbf","Type":"ContainerStarted","Data":"376eabfad4be616716b578f3918e6800b8fe2eb6a7ce7b84cb32e95810229ef3"} Oct 30 19:03:56.228771 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.228746 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" event={"ID":"bd7cda5e-1a4b-45f8-9cab-f96111986c08","Type":"ContainerStarted","Data":"1107f3e4351949dcfe0a9516bdcf58cd346932add8fa00e07c869b68efcea9f1"} Oct 30 19:03:56.241116 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.241062 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-fs89c" podStartSLOduration=400.241044473 podStartE2EDuration="6m40.241044473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:56.240360453 +0000 UTC m=+30.828265634" watchObservedRunningTime="2026-10-30 19:03:56.241044473 +0000 UTC m=+30.828949652" Oct 30 19:03:56.276111 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.276049 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-pqg4r" podStartSLOduration=400.276029259 podStartE2EDuration="6m40.276029259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:56.274211271 +0000 UTC m=+30.862116451" watchObservedRunningTime="2026-10-30 19:03:56.276029259 +0000 UTC m=+30.863934435" Oct 30 19:03:56.310294 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.309426 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-sfbbt" podStartSLOduration=400.309409668 podStartE2EDuration="6m40.309409668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:56.309187239 +0000 UTC m=+30.897092419" watchObservedRunningTime="2026-10-30 19:03:56.309409668 +0000 UTC m=+30.897314849" Oct 30 19:03:56.501229 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.501146 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:56.501229 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.501202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:03:56.501419 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.501268 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nxngq\" (UID: \"bcf8833f-ed61-40f4-880c-2645ffae454b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" Oct 30 19:03:56.501419 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.501324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:56.501419 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.501351 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:03:56.501419 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.501404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:56.501614 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.501479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:56.501665 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:56.501629 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle podName:20403b1e-e479-45b9-bb85-52cf10725d86 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:58.501608997 +0000 UTC m=+33.089514159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle") pod "router-default-77d478f789-d787c" (UID: "20403b1e-e479-45b9-bb85-52cf10725d86") : configmap references non-existent config key: service-ca.crt Oct 30 19:03:56.502095 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:56.502072 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:03:56.502208 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:56.502135 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs podName:20403b1e-e479-45b9-bb85-52cf10725d86 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:58.50211792 +0000 UTC m=+33.090023090 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs") pod "router-default-77d478f789-d787c" (UID: "20403b1e-e479-45b9-bb85-52cf10725d86") : secret "router-metrics-certs-default" not found Oct 30 19:03:56.502208 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:56.502195 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:03:56.502313 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:56.502230 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls podName:540944ec-348d-46e1-836b-9aeb07d16713 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:58.502219263 +0000 UTC m=+33.090124434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x4r6k" (UID: "540944ec-348d-46e1-836b-9aeb07d16713") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:03:56.502313 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:56.502279 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:03:56.502313 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:56.502308 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert podName:bcf8833f-ed61-40f4-880c-2645ffae454b nodeName:}" failed. No retries permitted until 2026-10-30 19:03:58.502298121 +0000 UTC m=+33.090203282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nxngq" (UID: "bcf8833f-ed61-40f4-880c-2645ffae454b") : secret "networking-console-plugin-cert" not found Oct 30 19:03:56.502461 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:56.502385 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:03:56.502461 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:56.502397 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4d8bb96d-62jtz: secret "image-registry-tls" not found Oct 30 19:03:56.502461 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:56.502441 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls podName:6af08cb7-fa60-491b-88fd-07c1ace45779 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:58.502430673 +0000 UTC m=+33.090335844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls") pod "image-registry-5d4d8bb96d-62jtz" (UID: "6af08cb7-fa60-491b-88fd-07c1ace45779") : secret "image-registry-tls" not found Oct 30 19:03:56.502593 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:56.502502 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:03:56.502593 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:56.502532 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls podName:b50175d2-e6bb-463a-9602-5bf9e4719a15 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:58.502522034 +0000 UTC m=+33.090427204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lx6v2" (UID: "b50175d2-e6bb-463a-9602-5bf9e4719a15") : secret "samples-operator-tls" not found Oct 30 19:03:56.502593 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:56.502589 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:03:56.502730 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:56.502599 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd9cf966f-mn64l: secret "image-registry-tls" not found Oct 30 19:03:56.502730 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:56.502628 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls podName:3ac3ab69-eead-4eca-af11-53fec376a6ac nodeName:}" failed. No retries permitted until 2026-10-30 19:03:58.50261663 +0000 UTC m=+33.090521799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls") pod "image-registry-5fd9cf966f-mn64l" (UID: "3ac3ab69-eead-4eca-af11-53fec376a6ac") : secret "image-registry-tls" not found Oct 30 19:03:56.602769 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:56.602378 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:03:56.602769 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:56.602663 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:03:56.602769 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:56.602724 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls podName:a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b nodeName:}" failed. No retries permitted until 2026-10-30 19:03:58.602706027 +0000 UTC m=+33.190611204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls") pod "dns-default-jqrww" (UID: "a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b") : secret "dns-default-metrics-tls" not found Oct 30 19:03:58.523968 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:58.523882 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:58.523968 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:58.523945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:03:58.524523 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:58.523979 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:03:58.524523 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:58.524066 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nxngq\" (UID: \"bcf8833f-ed61-40f4-880c-2645ffae454b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" Oct 30 19:03:58.524523 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:58.524120 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:03:58.524523 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:58.524147 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:03:58.524523 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:58.524202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:03:58.524523 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:58.524382 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:03:58.524523 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:58.524396 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd9cf966f-mn64l: secret "image-registry-tls" not found Oct 30 19:03:58.524523 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:58.524453 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls podName:3ac3ab69-eead-4eca-af11-53fec376a6ac nodeName:}" failed. No retries permitted until 2026-10-30 19:04:02.524435674 +0000 UTC m=+37.112340832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls") pod "image-registry-5fd9cf966f-mn64l" (UID: "3ac3ab69-eead-4eca-af11-53fec376a6ac") : secret "image-registry-tls" not found Oct 30 19:03:58.524523 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:58.524525 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle podName:20403b1e-e479-45b9-bb85-52cf10725d86 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:02.524515257 +0000 UTC m=+37.112420418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle") pod "router-default-77d478f789-d787c" (UID: "20403b1e-e479-45b9-bb85-52cf10725d86") : configmap references non-existent config key: service-ca.crt Oct 30 19:03:58.525087 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:58.524595 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:03:58.525087 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:58.524632 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs podName:20403b1e-e479-45b9-bb85-52cf10725d86 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:02.524622275 +0000 UTC m=+37.112527431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs") pod "router-default-77d478f789-d787c" (UID: "20403b1e-e479-45b9-bb85-52cf10725d86") : secret "router-metrics-certs-default" not found Oct 30 19:03:58.525087 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:58.524679 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:03:58.525087 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:58.524711 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls podName:540944ec-348d-46e1-836b-9aeb07d16713 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:02.52470128 +0000 UTC m=+37.112606435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x4r6k" (UID: "540944ec-348d-46e1-836b-9aeb07d16713") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:03:58.525087 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:58.524765 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:03:58.525087 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:58.524795 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert podName:bcf8833f-ed61-40f4-880c-2645ffae454b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:02.524785797 +0000 UTC m=+37.112690959 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nxngq" (UID: "bcf8833f-ed61-40f4-880c-2645ffae454b") : secret "networking-console-plugin-cert" not found Oct 30 19:03:58.525087 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:58.524855 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:03:58.525087 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:58.524865 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4d8bb96d-62jtz: secret "image-registry-tls" not found Oct 30 19:03:58.525087 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:58.524892 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls podName:6af08cb7-fa60-491b-88fd-07c1ace45779 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:02.52488366 +0000 UTC m=+37.112788815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls") pod "image-registry-5d4d8bb96d-62jtz" (UID: "6af08cb7-fa60-491b-88fd-07c1ace45779") : secret "image-registry-tls" not found Oct 30 19:03:58.525087 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:58.524931 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:03:58.525087 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:58.524961 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls podName:b50175d2-e6bb-463a-9602-5bf9e4719a15 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:02.52495168 +0000 UTC m=+37.112856846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lx6v2" (UID: "b50175d2-e6bb-463a-9602-5bf9e4719a15") : secret "samples-operator-tls" not found Oct 30 19:03:58.625069 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:58.625030 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:03:58.625340 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:58.625315 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:03:58.625465 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:58.625376 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls podName:a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:02.625360055 +0000 UTC m=+37.213265227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls") pod "dns-default-jqrww" (UID: "a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b") : secret "dns-default-metrics-tls" not found Oct 30 19:03:59.255781 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:59.255740 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jdc67" event={"ID":"969393ae-3d0e-4f83-bf03-0396c386ffbf","Type":"ContainerStarted","Data":"a1fb50260ba15b4a9e1aba6e01ae6283d67dc3153eeddf34d5e7f78a7282bd10"} Oct 30 19:03:59.291542 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:59.291087 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jdc67" podStartSLOduration=370.679629748 podStartE2EDuration="6m13.29106895s" totalImagesPullingTime="2.611439202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:59.28950571 +0000 UTC m=+33.877410890" watchObservedRunningTime="2026-10-30 19:03:59.29106895 +0000 UTC m=+33.878974130" Oct 30 19:03:59.735184 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:59.735100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:03:59.735184 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:59.735152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrxn\" (UniqueName: \"kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn\") pod \"network-check-target-zwhqt\" (UID: \"ee4d07b7-2acc-456e-932b-bfefb2f666df\") " pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:59.735687 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:59.735291 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:03:59.735687 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:03:59.735383 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs podName:93f0a70e-357e-4d65-b554-42363280bebd nodeName:}" failed. No retries permitted until 2026-10-30 19:04:31.735361783 +0000 UTC m=+66.323266942 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs") pod "network-metrics-daemon-frj4v" (UID: "93f0a70e-357e-4d65-b554-42363280bebd") : secret "metrics-daemon-secret" not found Oct 30 19:03:59.740210 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:59.740182 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftrxn\" (UniqueName: \"kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn\") pod \"network-check-target-zwhqt\" (UID: \"ee4d07b7-2acc-456e-932b-bfefb2f666df\") " pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:59.818872 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:59.818833 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:03:59.835662 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:59.835629 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:03:59.838658 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:03:59.838632 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:04:00.123473 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:00.123432 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:04:00.641189 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:00.641162 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zwhqt"] Oct 30 19:04:00.648798 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:00.648774 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hx495"] Oct 30 19:04:00.662875 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:04:00.662837 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4d07b7_2acc_456e_932b_bfefb2f666df.slice/crio-4b378dcf907d51ccab7982b3c15d671b97fae5af79e334b69b5b0594b5f61c0a WatchSource:0}: Error finding container 4b378dcf907d51ccab7982b3c15d671b97fae5af79e334b69b5b0594b5f61c0a: Status 404 returned error can't find the container with id 4b378dcf907d51ccab7982b3c15d671b97fae5af79e334b69b5b0594b5f61c0a Oct 30 19:04:00.663813 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:04:00.663782 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4cd78b_b491_4927_80ae_dae026d10e7a.slice/crio-e118e1579fbf034057e029e9bf4c55fc16e8477097cb06e1ed0245d2be15f79d WatchSource:0}: Error finding container e118e1579fbf034057e029e9bf4c55fc16e8477097cb06e1ed0245d2be15f79d: Status 404 returned error can't find the container with id e118e1579fbf034057e029e9bf4c55fc16e8477097cb06e1ed0245d2be15f79d Oct 30 19:04:01.265500 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.265409 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" event={"ID":"721cf657-76b1-4fdf-868d-d84cb9fa0b14","Type":"ContainerStarted","Data":"1bbf3f90a4c1ebd958b962674daf83104910396b0d3cf3e4678b3fb830e86bd6"} Oct 30 19:04:01.266996 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.266968 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" event={"ID":"c6188415-c8d5-4972-9d6d-f6e3bcf4b079","Type":"ContainerStarted","Data":"156210b86be7e2ecd7c970bad403ffa89797701f1905637ce60f467218bb38d7"} Oct 30 19:04:01.268156 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.268136 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zwhqt" event={"ID":"ee4d07b7-2acc-456e-932b-bfefb2f666df","Type":"ContainerStarted","Data":"4b378dcf907d51ccab7982b3c15d671b97fae5af79e334b69b5b0594b5f61c0a"} Oct 30 19:04:01.269515 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.269494 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ccd7w" event={"ID":"3e626b03-28b0-4840-bbe2-518834311815","Type":"ContainerStarted","Data":"4155cd2eb5bf266c4d08e8895e12086c63d53f54c60bbaeea209e5b5d688b82f"} Oct 30 19:04:01.271069 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.271047 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wd8lm" event={"ID":"11d31a53-ae1b-4d46-aa8c-6c00e89ed539","Type":"ContainerStarted","Data":"81c26b6e909fc8f6eedbfe63cb64a1fe1e7a5f03c1e96645094aaf67c2c245d0"} Oct 30 19:04:01.272258 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.272234 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hx495" event={"ID":"ad4cd78b-b491-4927-80ae-dae026d10e7a","Type":"ContainerStarted","Data":"e118e1579fbf034057e029e9bf4c55fc16e8477097cb06e1ed0245d2be15f79d"} Oct 30 19:04:01.273746 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.273722 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cfdmh" event={"ID":"bfcde492-9464-42cc-a01c-84d9eec4135a","Type":"ContainerStarted","Data":"8ec048b8c3dc811fc1a2e0871583bb87306cc22a0e845a3b3c0c20de0a6f97e3"} Oct 30 19:04:01.275917 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.275898 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/0.log" Oct 30 19:04:01.275994 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.275938 2574 generic.go:356] "Generic (PLEG): container finished" podID="bd7cda5e-1a4b-45f8-9cab-f96111986c08" containerID="f6b9c297ace4ab84d1ec150c454f184cbfa0d7a9b1d71a56d8e907e93edcf899" exitCode=255 Oct 30 19:04:01.275994 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.275967 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" event={"ID":"bd7cda5e-1a4b-45f8-9cab-f96111986c08","Type":"ContainerDied","Data":"f6b9c297ace4ab84d1ec150c454f184cbfa0d7a9b1d71a56d8e907e93edcf899"} Oct 30 19:04:01.276155 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.276141 2574 scope.go:122] "RemoveContainer" containerID="f6b9c297ace4ab84d1ec150c454f184cbfa0d7a9b1d71a56d8e907e93edcf899" Oct 30 19:04:01.300772 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.300720 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" podStartSLOduration=370.459763889 podStartE2EDuration="6m15.300701153s" totalImagesPullingTime="4.840937264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:01.300391195 +0000 UTC m=+35.888296373" watchObservedRunningTime="2026-10-30 19:04:01.300701153 +0000 UTC m=+35.888606331" Oct 30 19:04:01.370577 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.369666 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" podStartSLOduration=370.493973699 podStartE2EDuration="6m15.369647697s" totalImagesPullingTime="4.875673998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:01.368694512 +0000 UTC m=+35.956599690" watchObservedRunningTime="2026-10-30 19:04:01.369647697 +0000 UTC m=+35.957552875" Oct 30 19:04:01.411921 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.411851 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-wd8lm" podStartSLOduration=370.597660003 podStartE2EDuration="6m15.411831871s" totalImagesPullingTime="4.814171868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:01.407606782 +0000 UTC m=+35.995511972" watchObservedRunningTime="2026-10-30 19:04:01.411831871 +0000 UTC m=+35.999737050" Oct 30 19:04:01.412803 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.412779 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6zk7f"] Oct 30 19:04:01.433440 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.433396 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lp2xk"] Oct 30 19:04:01.434543 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.434521 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6zk7f" Oct 30 19:04:01.450322 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.450292 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6zk7f"] Oct 30 19:04:01.450483 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.450371 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lp2xk"] Oct 30 19:04:01.450544 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.450479 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lp2xk" Oct 30 19:04:01.451952 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.451927 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m72r5\" (UniqueName: \"kubernetes.io/projected/facfc926-9288-41c4-bdc3-5d8acf446292-kube-api-access-m72r5\") pod \"migrator-66df5d496b-lp2xk\" (UID: \"facfc926-9288-41c4-bdc3-5d8acf446292\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lp2xk" Oct 30 19:04:01.452157 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.452137 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlw9l\" (UniqueName: \"kubernetes.io/projected/452b3edc-e035-446c-9b7b-ee8154f3c6c3-kube-api-access-vlw9l\") pod \"migrator-66df5d496b-6zk7f\" (UID: \"452b3edc-e035-446c-9b7b-ee8154f3c6c3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6zk7f" Oct 30 19:04:01.455381 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.455359 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9ph75\"" Oct 30 19:04:01.464173 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.464121 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 19:04:01.464455 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.464436 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 19:04:01.520935 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.520848 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cfdmh" podStartSLOduration=370.160170677 podStartE2EDuration="6m15.520834764s" totalImagesPullingTime="5.360664087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:01.482787835 +0000 UTC m=+36.070693014" watchObservedRunningTime="2026-10-30 19:04:01.520834764 +0000 UTC m=+36.108739942" Oct 30 19:04:01.521101 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.520949 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ccd7w" podStartSLOduration=2.194849921 podStartE2EDuration="7.520942937s" totalImagesPullingTime="5.326093016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:01.519080241 +0000 UTC m=+36.106985420" watchObservedRunningTime="2026-10-30 19:04:01.520942937 +0000 UTC m=+36.108848112" Oct 30 19:04:01.548631 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.548598 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4946j"] Oct 30 19:04:01.553307 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.553269 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m72r5\" (UniqueName: \"kubernetes.io/projected/facfc926-9288-41c4-bdc3-5d8acf446292-kube-api-access-m72r5\") pod \"migrator-66df5d496b-lp2xk\" (UID: \"facfc926-9288-41c4-bdc3-5d8acf446292\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lp2xk" Oct 30 19:04:01.553526 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.553507 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlw9l\" (UniqueName: \"kubernetes.io/projected/452b3edc-e035-446c-9b7b-ee8154f3c6c3-kube-api-access-vlw9l\") pod \"migrator-66df5d496b-6zk7f\" (UID: \"452b3edc-e035-446c-9b7b-ee8154f3c6c3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6zk7f" Oct 30 19:04:01.562745 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.562722 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" Oct 30 19:04:01.568686 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.568652 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4946j"] Oct 30 19:04:01.576266 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.576242 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 19:04:01.577603 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.577385 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 19:04:01.577886 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.577870 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-h2r74\"" Oct 30 19:04:01.577964 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.577958 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 19:04:01.578060 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.578045 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 19:04:01.580160 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.580108 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlw9l\" (UniqueName: \"kubernetes.io/projected/452b3edc-e035-446c-9b7b-ee8154f3c6c3-kube-api-access-vlw9l\") pod \"migrator-66df5d496b-6zk7f\" (UID: \"452b3edc-e035-446c-9b7b-ee8154f3c6c3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6zk7f" Oct 30 19:04:01.582116 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.582071 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m72r5\" (UniqueName: \"kubernetes.io/projected/facfc926-9288-41c4-bdc3-5d8acf446292-kube-api-access-m72r5\") pod \"migrator-66df5d496b-lp2xk\" (UID: \"facfc926-9288-41c4-bdc3-5d8acf446292\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lp2xk" Oct 30 19:04:01.648956 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.647593 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5rvfx"] Oct 30 19:04:01.654063 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.654034 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/68061b2f-50b6-456d-86e7-4b766c7925bc-signing-key\") pod \"service-ca-5fffc97bdf-4946j\" (UID: \"68061b2f-50b6-456d-86e7-4b766c7925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" Oct 30 19:04:01.654197 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.654097 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlsmw\" (UniqueName: \"kubernetes.io/projected/68061b2f-50b6-456d-86e7-4b766c7925bc-kube-api-access-qlsmw\") pod \"service-ca-5fffc97bdf-4946j\" (UID: \"68061b2f-50b6-456d-86e7-4b766c7925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" Oct 30 19:04:01.654294 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.654274 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/68061b2f-50b6-456d-86e7-4b766c7925bc-signing-cabundle\") pod \"service-ca-5fffc97bdf-4946j\" (UID: \"68061b2f-50b6-456d-86e7-4b766c7925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" Oct 30 19:04:01.669572 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.669538 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:01.673090 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.673052 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5rvfx"] Oct 30 19:04:01.681464 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.681431 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:04:01.681587 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.681509 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:04:01.681587 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.681534 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pbgr5\"" Oct 30 19:04:01.698726 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.698696 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-hkjk5-l9rq7"] Oct 30 19:04:01.713453 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.713428 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:01.719314 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.719249 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hkjk5-l9rq7"] Oct 30 19:04:01.755614 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.755580 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-trusted-ca-bundle\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:01.755764 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.755682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/68061b2f-50b6-456d-86e7-4b766c7925bc-signing-key\") pod \"service-ca-5fffc97bdf-4946j\" (UID: \"68061b2f-50b6-456d-86e7-4b766c7925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" Oct 30 19:04:01.755764 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.755747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlsmw\" (UniqueName: \"kubernetes.io/projected/68061b2f-50b6-456d-86e7-4b766c7925bc-kube-api-access-qlsmw\") pod \"service-ca-5fffc97bdf-4946j\" (UID: \"68061b2f-50b6-456d-86e7-4b766c7925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" Oct 30 19:04:01.755831 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.755791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-service-ca-bundle\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:01.755831 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.755807 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmxmw\" (UniqueName: \"kubernetes.io/projected/1a7defa5-7e66-4e17-8dc8-e74137e4f054-kube-api-access-gmxmw\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:01.755991 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.755973 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5557e025-30b7-4315-b7a1-308e166c9a3b-data-volume\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:01.756070 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.756056 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1a7defa5-7e66-4e17-8dc8-e74137e4f054-archives-path\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:01.756120 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.756105 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4jzx\" (UniqueName: \"kubernetes.io/projected/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-api-access-j4jzx\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:01.756163 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.756132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/68061b2f-50b6-456d-86e7-4b766c7925bc-signing-cabundle\") pod \"service-ca-5fffc97bdf-4946j\" (UID: \"68061b2f-50b6-456d-86e7-4b766c7925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" Oct 30 19:04:01.756206 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.756179 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:01.756295 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.756276 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5557e025-30b7-4315-b7a1-308e166c9a3b-crio-socket\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:01.756364 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.756334 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:01.756802 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.756786 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/68061b2f-50b6-456d-86e7-4b766c7925bc-signing-cabundle\") pod \"service-ca-5fffc97bdf-4946j\" (UID: \"68061b2f-50b6-456d-86e7-4b766c7925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" Oct 30 19:04:01.758490 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.758446 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/68061b2f-50b6-456d-86e7-4b766c7925bc-signing-key\") pod \"service-ca-5fffc97bdf-4946j\" (UID: \"68061b2f-50b6-456d-86e7-4b766c7925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" Oct 30 19:04:01.772150 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.772097 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6zk7f" Oct 30 19:04:01.773034 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.772995 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlsmw\" (UniqueName: \"kubernetes.io/projected/68061b2f-50b6-456d-86e7-4b766c7925bc-kube-api-access-qlsmw\") pod \"service-ca-5fffc97bdf-4946j\" (UID: \"68061b2f-50b6-456d-86e7-4b766c7925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" Oct 30 19:04:01.853706 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.853673 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lp2xk" Oct 30 19:04:01.859888 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.857597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-trusted-ca-bundle\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:01.859888 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.857744 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-service-ca-bundle\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:01.859888 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.857769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmxmw\" (UniqueName: \"kubernetes.io/projected/1a7defa5-7e66-4e17-8dc8-e74137e4f054-kube-api-access-gmxmw\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:01.859888 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.857835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5557e025-30b7-4315-b7a1-308e166c9a3b-data-volume\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:01.859888 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.857876 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1a7defa5-7e66-4e17-8dc8-e74137e4f054-archives-path\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:01.859888 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.857924 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4jzx\" (UniqueName: \"kubernetes.io/projected/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-api-access-j4jzx\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:01.859888 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.857957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:01.859888 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.857986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5557e025-30b7-4315-b7a1-308e166c9a3b-crio-socket\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:01.859888 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.858023 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:01.859888 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:01.858144 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:04:01.859888 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:01.858207 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-rbac-proxy-cm podName:5557e025-30b7-4315-b7a1-308e166c9a3b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:02.358187338 +0000 UTC m=+36.946092501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5rvfx" (UID: "5557e025-30b7-4315-b7a1-308e166c9a3b") : configmap "kube-rbac-proxy" not found Oct 30 19:04:01.859888 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.859099 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-trusted-ca-bundle\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:01.859888 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.859415 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-service-ca-bundle\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:01.859888 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:01.859803 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:04:01.859888 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.859822 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5557e025-30b7-4315-b7a1-308e166c9a3b-crio-socket\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:01.859888 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:01.859858 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls podName:5557e025-30b7-4315-b7a1-308e166c9a3b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:02.359839949 +0000 UTC m=+36.947745120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rvfx" (UID: "5557e025-30b7-4315-b7a1-308e166c9a3b") : secret "insights-runtime-extractor-tls" not found Oct 30 19:04:01.860762 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.860118 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1a7defa5-7e66-4e17-8dc8-e74137e4f054-archives-path\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:01.861115 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.861081 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5557e025-30b7-4315-b7a1-308e166c9a3b-data-volume\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:01.877196 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.877171 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmxmw\" (UniqueName: \"kubernetes.io/projected/1a7defa5-7e66-4e17-8dc8-e74137e4f054-kube-api-access-gmxmw\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:01.877306 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.877254 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4jzx\" (UniqueName: \"kubernetes.io/projected/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-api-access-j4jzx\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:01.900264 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:01.900232 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" Oct 30 19:04:02.045051 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.044951 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:02.090218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.089822 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6zk7f"] Oct 30 19:04:02.093057 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:04:02.092999 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452b3edc_e035_446c_9b7b_ee8154f3c6c3.slice/crio-7a6e72a2cf13bb7fba89ef5293f61f7f1ee7ab4085817a8f80353dd02bae8b8a WatchSource:0}: Error finding container 7a6e72a2cf13bb7fba89ef5293f61f7f1ee7ab4085817a8f80353dd02bae8b8a: Status 404 returned error can't find the container with id 7a6e72a2cf13bb7fba89ef5293f61f7f1ee7ab4085817a8f80353dd02bae8b8a Oct 30 19:04:02.183765 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.183738 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lp2xk"] Oct 30 19:04:02.186570 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:04:02.186538 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfacfc926_9288_41c4_bdc3_5d8acf446292.slice/crio-df0274035fb0fb4f6a09a0d067b69f2f73173263d4f4e814bd26e550a97f566e WatchSource:0}: Error finding container df0274035fb0fb4f6a09a0d067b69f2f73173263d4f4e814bd26e550a97f566e: Status 404 returned error can't find the container with id df0274035fb0fb4f6a09a0d067b69f2f73173263d4f4e814bd26e550a97f566e Oct 30 19:04:02.222284 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:04:02.222247 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68061b2f_50b6_456d_86e7_4b766c7925bc.slice/crio-400470efa985001a73273717eec4096cb361f495ba0b0e9aeaba126ebb524a63 WatchSource:0}: Error finding container 400470efa985001a73273717eec4096cb361f495ba0b0e9aeaba126ebb524a63: Status 404 returned error can't find the container with id 400470efa985001a73273717eec4096cb361f495ba0b0e9aeaba126ebb524a63 Oct 30 19:04:02.222759 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.222712 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4946j"] Oct 30 19:04:02.282684 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.282657 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/1.log" Oct 30 19:04:02.283199 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.283178 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/0.log" Oct 30 19:04:02.283268 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.283213 2574 generic.go:356] "Generic (PLEG): container finished" podID="bd7cda5e-1a4b-45f8-9cab-f96111986c08" containerID="33635e76a40848e3ec3bc6c36890b08ed835462c6bdf12fcb8d4bee6498cbb95" exitCode=255 Oct 30 19:04:02.283316 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.283303 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" event={"ID":"bd7cda5e-1a4b-45f8-9cab-f96111986c08","Type":"ContainerDied","Data":"33635e76a40848e3ec3bc6c36890b08ed835462c6bdf12fcb8d4bee6498cbb95"} Oct 30 19:04:02.283378 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.283368 2574 scope.go:122] "RemoveContainer" containerID="f6b9c297ace4ab84d1ec150c454f184cbfa0d7a9b1d71a56d8e907e93edcf899" Oct 30 19:04:02.283763 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.283745 2574 scope.go:122] "RemoveContainer" containerID="33635e76a40848e3ec3bc6c36890b08ed835462c6bdf12fcb8d4bee6498cbb95" Oct 30 19:04:02.283991 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.283966 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6h7t5_openshift-console-operator(bd7cda5e-1a4b-45f8-9cab-f96111986c08)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" podUID="bd7cda5e-1a4b-45f8-9cab-f96111986c08" Oct 30 19:04:02.287080 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.286721 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zwhqt" event={"ID":"ee4d07b7-2acc-456e-932b-bfefb2f666df","Type":"ContainerStarted","Data":"40cbe70929182b9543405095edb1e3a8374a94a9aa72acc176194e083d33b92b"} Oct 30 19:04:02.287080 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.286789 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:04:02.288280 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.288255 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" event={"ID":"68061b2f-50b6-456d-86e7-4b766c7925bc","Type":"ContainerStarted","Data":"400470efa985001a73273717eec4096cb361f495ba0b0e9aeaba126ebb524a63"} Oct 30 19:04:02.289617 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.289592 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lp2xk" event={"ID":"facfc926-9288-41c4-bdc3-5d8acf446292","Type":"ContainerStarted","Data":"df0274035fb0fb4f6a09a0d067b69f2f73173263d4f4e814bd26e550a97f566e"} Oct 30 19:04:02.290871 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.290845 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6zk7f" event={"ID":"452b3edc-e035-446c-9b7b-ee8154f3c6c3","Type":"ContainerStarted","Data":"7a6e72a2cf13bb7fba89ef5293f61f7f1ee7ab4085817a8f80353dd02bae8b8a"} Oct 30 19:04:02.357218 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.356791 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zwhqt" podStartSLOduration=35.787948423 podStartE2EDuration="36.356771979s" totalImagesPullingTime="568.823556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:02.354029585 +0000 UTC m=+36.941934764" watchObservedRunningTime="2026-10-30 19:04:02.356771979 +0000 UTC m=+36.944677152" Oct 30 19:04:02.364055 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.364021 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:02.364203 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.364102 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:04:02.364203 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.364176 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-rbac-proxy-cm podName:5557e025-30b7-4315-b7a1-308e166c9a3b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:03.364156907 +0000 UTC m=+37.952062077 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5rvfx" (UID: "5557e025-30b7-4315-b7a1-308e166c9a3b") : configmap "kube-rbac-proxy" not found Oct 30 19:04:02.364941 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.364908 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:02.365250 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.365229 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:04:02.365339 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.365286 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls podName:5557e025-30b7-4315-b7a1-308e166c9a3b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:03.36526862 +0000 UTC m=+37.953173793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rvfx" (UID: "5557e025-30b7-4315-b7a1-308e166c9a3b") : secret "insights-runtime-extractor-tls" not found Oct 30 19:04:02.370837 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.370809 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hkjk5-l9rq7"] Oct 30 19:04:02.379676 ip-10-0-140-235 kubenswrapper[2574]: W1030 19:04:02.379646 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7defa5_7e66_4e17_8dc8_e74137e4f054.slice/crio-235840f48c7642fabfb8577e4dc8aae46f92e2b241626f9eda4b30ae41a26152 WatchSource:0}: Error finding container 235840f48c7642fabfb8577e4dc8aae46f92e2b241626f9eda4b30ae41a26152: Status 404 returned error can't find the container with id 235840f48c7642fabfb8577e4dc8aae46f92e2b241626f9eda4b30ae41a26152 Oct 30 19:04:02.566714 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.566670 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:02.566868 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.566722 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:02.566868 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.566749 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:04:02.566868 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.566807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nxngq\" (UID: \"bcf8833f-ed61-40f4-880c-2645ffae454b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" Oct 30 19:04:02.566868 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.566844 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle podName:20403b1e-e479-45b9-bb85-52cf10725d86 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:10.566823619 +0000 UTC m=+45.154728779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle") pod "router-default-77d478f789-d787c" (UID: "20403b1e-e479-45b9-bb85-52cf10725d86") : configmap references non-existent config key: service-ca.crt Oct 30 19:04:02.567053 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.566860 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:04:02.567053 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.566878 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:04:02.567053 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.566895 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:02.567053 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.566860 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:04:02.567053 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.566916 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert podName:bcf8833f-ed61-40f4-880c-2645ffae454b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:10.566905813 +0000 UTC m=+45.154810969 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nxngq" (UID: "bcf8833f-ed61-40f4-880c-2645ffae454b") : secret "networking-console-plugin-cert" not found Oct 30 19:04:02.567053 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.566950 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls podName:540944ec-348d-46e1-836b-9aeb07d16713 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:10.566941331 +0000 UTC m=+45.154846498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x4r6k" (UID: "540944ec-348d-46e1-836b-9aeb07d16713") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:04:02.567053 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.566963 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:04:02.567053 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.566969 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs podName:20403b1e-e479-45b9-bb85-52cf10725d86 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:10.566960656 +0000 UTC m=+45.154865819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs") pod "router-default-77d478f789-d787c" (UID: "20403b1e-e479-45b9-bb85-52cf10725d86") : secret "router-metrics-certs-default" not found Oct 30 19:04:02.567053 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.566975 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4d8bb96d-62jtz: secret "image-registry-tls" not found Oct 30 19:04:02.567053 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.566993 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:04:02.567053 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.567021 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls podName:6af08cb7-fa60-491b-88fd-07c1ace45779 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:10.56699625 +0000 UTC m=+45.154901422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls") pod "image-registry-5d4d8bb96d-62jtz" (UID: "6af08cb7-fa60-491b-88fd-07c1ace45779") : secret "image-registry-tls" not found Oct 30 19:04:02.567442 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.567069 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:04:02.567442 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.567103 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls podName:b50175d2-e6bb-463a-9602-5bf9e4719a15 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:10.567092967 +0000 UTC m=+45.154998123 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lx6v2" (UID: "b50175d2-e6bb-463a-9602-5bf9e4719a15") : secret "samples-operator-tls" not found Oct 30 19:04:02.567442 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.567068 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:02.567442 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.567123 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:04:02.567442 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.567134 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd9cf966f-mn64l: secret "image-registry-tls" not found Oct 30 19:04:02.567442 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.567167 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls podName:3ac3ab69-eead-4eca-af11-53fec376a6ac nodeName:}" failed. No retries permitted until 2026-10-30 19:04:10.567157872 +0000 UTC m=+45.155063042 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls") pod "image-registry-5fd9cf966f-mn64l" (UID: "3ac3ab69-eead-4eca-af11-53fec376a6ac") : secret "image-registry-tls" not found Oct 30 19:04:02.667975 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:02.667937 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:04:02.668184 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.668123 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:04:02.668249 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:02.668191 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls podName:a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:10.668172949 +0000 UTC m=+45.256078105 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls") pod "dns-default-jqrww" (UID: "a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b") : secret "dns-default-metrics-tls" not found Oct 30 19:04:03.320076 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:03.319786 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" event={"ID":"1a7defa5-7e66-4e17-8dc8-e74137e4f054","Type":"ContainerStarted","Data":"fb31b16220f88daec24acd71d67fb60a326a3b18f89e3de1b3d895fd97bbd3fb"} Oct 30 19:04:03.320076 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:03.319847 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" event={"ID":"1a7defa5-7e66-4e17-8dc8-e74137e4f054","Type":"ContainerStarted","Data":"235840f48c7642fabfb8577e4dc8aae46f92e2b241626f9eda4b30ae41a26152"} Oct 30 19:04:03.325580 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:03.324780 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/1.log" Oct 30 19:04:03.325580 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:03.325135 2574 scope.go:122] "RemoveContainer" containerID="33635e76a40848e3ec3bc6c36890b08ed835462c6bdf12fcb8d4bee6498cbb95" Oct 30 19:04:03.325580 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:03.325300 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6h7t5_openshift-console-operator(bd7cda5e-1a4b-45f8-9cab-f96111986c08)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" podUID="bd7cda5e-1a4b-45f8-9cab-f96111986c08" Oct 30 19:04:03.328026 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:03.327957 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" event={"ID":"68061b2f-50b6-456d-86e7-4b766c7925bc","Type":"ContainerStarted","Data":"1eedc95ffd77a9297a6e0d11ddd45a775fa4e28b731d2722b5f31a7431500ad7"} Oct 30 19:04:03.377513 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:03.377474 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:03.377683 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:03.377525 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:03.380580 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:03.379171 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:04:03.380580 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:03.379207 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:04:03.380580 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:03.379242 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-rbac-proxy-cm podName:5557e025-30b7-4315-b7a1-308e166c9a3b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:05.379223069 +0000 UTC m=+39.967128234 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5rvfx" (UID: "5557e025-30b7-4315-b7a1-308e166c9a3b") : configmap "kube-rbac-proxy" not found Oct 30 19:04:03.380580 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:03.379273 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls podName:5557e025-30b7-4315-b7a1-308e166c9a3b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:05.379255081 +0000 UTC m=+39.967160252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rvfx" (UID: "5557e025-30b7-4315-b7a1-308e166c9a3b") : secret "insights-runtime-extractor-tls" not found Oct 30 19:04:03.412089 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:03.410706 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" podStartSLOduration=2.410683249 podStartE2EDuration="2.410683249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:03.355475882 +0000 UTC m=+37.943381064" watchObservedRunningTime="2026-10-30 19:04:03.410683249 +0000 UTC m=+37.998588428" Oct 30 19:04:03.452854 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:03.452398 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" podStartSLOduration=2.452380309 podStartE2EDuration="2.452380309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:03.411930905 +0000 UTC m=+37.999836088" watchObservedRunningTime="2026-10-30 19:04:03.452380309 +0000 UTC m=+38.040285490" Oct 30 19:04:05.184305 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:05.184264 2574 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:04:05.184803 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:05.184372 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:04:05.184862 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:05.184823 2574 scope.go:122] "RemoveContainer" containerID="33635e76a40848e3ec3bc6c36890b08ed835462c6bdf12fcb8d4bee6498cbb95" Oct 30 19:04:05.185101 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:05.185076 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6h7t5_openshift-console-operator(bd7cda5e-1a4b-45f8-9cab-f96111986c08)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" podUID="bd7cda5e-1a4b-45f8-9cab-f96111986c08" Oct 30 19:04:05.341606 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:05.341566 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lp2xk" event={"ID":"facfc926-9288-41c4-bdc3-5d8acf446292","Type":"ContainerStarted","Data":"f8c8a954f2ba14473e2cc967f6b566695e575a9316ff47c193d514024c5c74fd"} Oct 30 19:04:05.343397 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:05.343364 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6zk7f" event={"ID":"452b3edc-e035-446c-9b7b-ee8154f3c6c3","Type":"ContainerStarted","Data":"e5e857b88117d89bf497bd56d16e64a4675893870abfdfe53b42e8c2c72ce1b1"} Oct 30 19:04:05.375481 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:05.375417 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lp2xk" podStartSLOduration=2.266931584 podStartE2EDuration="4.375399143s" totalImagesPullingTime="2.108467559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:05.374628389 +0000 UTC m=+39.962533567" watchObservedRunningTime="2026-10-30 19:04:05.375399143 +0000 UTC m=+39.963304324" Oct 30 19:04:05.401694 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:05.401660 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:05.401694 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:05.401699 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:05.401884 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:05.401820 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:04:05.401884 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:05.401865 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-rbac-proxy-cm podName:5557e025-30b7-4315-b7a1-308e166c9a3b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:09.401853732 +0000 UTC m=+43.989758888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5rvfx" (UID: "5557e025-30b7-4315-b7a1-308e166c9a3b") : configmap "kube-rbac-proxy" not found Oct 30 19:04:05.402906 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:05.402884 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:04:05.403028 ip-10-0-140-235 kubenswrapper[2574]: E1030 19:04:05.402931 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls podName:5557e025-30b7-4315-b7a1-308e166c9a3b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:09.402918047 +0000 UTC m=+43.990823203 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rvfx" (UID: "5557e025-30b7-4315-b7a1-308e166c9a3b") : secret "insights-runtime-extractor-tls" not found Oct 30 19:04:05.409524 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:05.409473 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6zk7f" podStartSLOduration=2.205169371 podStartE2EDuration="4.409456057s" totalImagesPullingTime="2.204286686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:05.407840156 +0000 UTC m=+39.995745336" watchObservedRunningTime="2026-10-30 19:04:05.409456057 +0000 UTC m=+39.997361236" Oct 30 19:04:06.841070 ip-10-0-140-235 kubenswrapper[2574]: I1030 19:04:06.840889 2574 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:04:06.841039 ip-10-0-140-235 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 19:04:06.862472 ip-10-0-140-235 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 19:04:06.862690 ip-10-0-140-235 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 19:04:06.862904 ip-10-0-140-235 systemd[1]: kubelet.service: Consumed 3.213s CPU time. Oct 30 19:04:06.882534 ip-10-0-140-235 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:04:06.957395 ip-10-0-140-235 kubenswrapper[5047]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:04:06.957395 ip-10-0-140-235 kubenswrapper[5047]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:04:06.957395 ip-10-0-140-235 kubenswrapper[5047]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:04:06.957395 ip-10-0-140-235 kubenswrapper[5047]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:04:06.959501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959410 5047 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:04:06.959501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959499 5047 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:04:06.959501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959503 5047 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:04:06.959501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959506 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959509 5047 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959512 5047 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959515 5047 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959518 5047 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959522 5047 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959527 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959530 5047 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959534 5047 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959536 5047 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959539 5047 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959542 5047 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959545 5047 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959548 5047 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959551 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959554 5047 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959557 5047 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959559 5047 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959562 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:04:06.959697 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959564 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959567 5047 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959570 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959573 5047 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959576 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959578 5047 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959581 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959584 5047 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959588 5047 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959591 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959593 5047 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959596 5047 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959599 5047 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959601 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959604 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959607 5047 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959610 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959613 5047 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959615 5047 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959618 5047 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:04:06.960515 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959620 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959623 5047 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959626 5047 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959628 5047 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959631 5047 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959633 5047 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959637 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959639 5047 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959643 5047 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959645 5047 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959648 5047 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959651 5047 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959653 5047 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959656 5047 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959659 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959661 5047 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959664 5047 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959667 5047 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959670 5047 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959672 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:04:06.961355 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959675 5047 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959677 5047 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959680 5047 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959683 5047 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959685 5047 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959688 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959691 5047 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959694 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959696 5047 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959699 5047 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959702 5047 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959705 5047 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959708 5047 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959710 5047 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959713 5047 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959716 5047 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959719 5047 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959721 5047 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959724 5047 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959728 5047 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:04:06.962182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959731 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959733 5047 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959736 5047 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959739 5047 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959741 5047 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959744 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959746 5047 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959749 5047 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959752 5047 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959755 5047 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959757 5047 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959760 5047 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959763 5047 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959765 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959768 5047 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959771 5047 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959775 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959777 5047 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959780 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:04:06.962999 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959785 5047 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959790 5047 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959793 5047 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959796 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.959799 5047 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960632 5047 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960641 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960644 5047 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960647 5047 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960649 5047 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960652 5047 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960655 5047 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960657 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960666 5047 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960669 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960672 5047 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960675 5047 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960678 5047 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960681 5047 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960684 5047 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:04:06.963799 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960686 5047 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960689 5047 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960692 5047 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960694 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960697 5047 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960700 5047 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960702 5047 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960705 5047 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960708 5047 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960711 5047 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960714 5047 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960716 5047 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960719 5047 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960722 5047 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960724 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960727 5047 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960729 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960732 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960735 5047 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:04:06.964662 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960738 5047 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960740 5047 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960743 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960745 5047 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960748 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960751 5047 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960759 5047 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960763 5047 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960767 5047 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960771 5047 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960774 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960777 5047 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960780 5047 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960782 5047 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960785 5047 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960788 5047 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960790 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960793 5047 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960796 5047 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:04:06.965462 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960798 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960801 5047 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960804 5047 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960809 5047 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960813 5047 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960816 5047 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960819 5047 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960821 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960824 5047 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960827 5047 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960830 5047 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960832 5047 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960835 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960837 5047 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960840 5047 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960842 5047 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960845 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960848 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960850 5047 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:04:06.966319 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960854 5047 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960857 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960860 5047 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960863 5047 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960865 5047 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960868 5047 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960870 5047 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960873 5047 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960876 5047 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960878 5047 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960881 5047 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960884 5047 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960887 5047 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960889 5047 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960892 5047 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960896 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960899 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960901 5047 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960904 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960907 5047 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:04:06.967123 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960909 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960912 5047 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960914 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960917 5047 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960919 5047 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960922 5047 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960925 5047 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960927 5047 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960930 5047 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960933 5047 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960936 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960939 5047 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960943 5047 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.960945 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961445 5047 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961450 5047 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961454 5047 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961456 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961459 5047 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961462 5047 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:04:06.967948 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961464 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961467 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961470 5047 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961473 5047 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961476 5047 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961478 5047 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961481 5047 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961486 5047 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961488 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961491 5047 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961493 5047 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961496 5047 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961499 5047 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961502 5047 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961504 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961507 5047 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961509 5047 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961512 5047 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961515 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961518 5047 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:04:06.968829 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961520 5047 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961523 5047 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961525 5047 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961528 5047 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961533 5047 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961536 5047 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961539 5047 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961542 5047 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961545 5047 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961548 5047 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961550 5047 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961553 5047 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961555 5047 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961558 5047 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961561 5047 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961563 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961566 5047 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961568 5047 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961571 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961576 5047 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:04:06.969660 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961579 5047 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961581 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961584 5047 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961587 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961591 5047 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961595 5047 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961598 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961601 5047 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961605 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961608 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961610 5047 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961614 5047 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961618 5047 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961622 5047 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961625 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961627 5047 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961632 5047 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961634 5047 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961638 5047 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:04:06.970501 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961640 5047 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961643 5047 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961646 5047 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961649 5047 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961652 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961654 5047 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961657 5047 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961660 5047 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961662 5047 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961665 5047 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961667 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961670 5047 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961674 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961676 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961679 5047 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961681 5047 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961684 5047 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961686 5047 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961690 5047 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961693 5047 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:04:06.971280 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961695 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961698 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961700 5047 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961703 5047 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961706 5047 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961708 5047 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961711 5047 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961713 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961716 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961720 5047 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961722 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961725 5047 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961727 5047 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961730 5047 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961733 5047 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961736 5047 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961739 5047 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961742 5047 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961744 5047 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:04:06.972189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961747 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.961750 5047 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961851 5047 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961861 5047 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961870 5047 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961880 5047 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961885 5047 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961888 5047 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961893 5047 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961897 5047 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961901 5047 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961904 5047 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961908 5047 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961911 5047 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961914 5047 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961917 5047 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961920 5047 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961924 5047 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961927 5047 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961930 5047 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961935 5047 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961938 5047 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961943 5047 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961946 5047 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:04:06.972973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961951 5047 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961955 5047 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961958 5047 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961961 5047 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961968 5047 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961971 5047 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961975 5047 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961978 5047 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961981 5047 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961984 5047 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961989 5047 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961992 5047 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961995 5047 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.961999 5047 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962014 5047 flags.go:64] FLAG: --enable-server="true" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962018 5047 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962022 5047 flags.go:64] FLAG: --event-burst="100" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962025 5047 flags.go:64] FLAG: --event-qps="50" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962028 5047 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962032 5047 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962036 5047 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962040 5047 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962043 5047 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962047 5047 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962050 5047 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:04:06.973961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962054 5047 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962057 5047 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962060 5047 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962063 5047 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962066 5047 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962070 5047 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962073 5047 flags.go:64] FLAG: --feature-gates="" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962078 5047 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962081 5047 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962084 5047 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962088 5047 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962091 5047 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962095 5047 flags.go:64] FLAG: --help="false" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962098 5047 flags.go:64] FLAG: --hostname-override="ip-10-0-140-235.ec2.internal" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962101 5047 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962105 5047 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962108 5047 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962111 5047 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962114 5047 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962118 5047 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962122 5047 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962125 5047 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962128 5047 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:04:06.974997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962131 5047 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962134 5047 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962137 5047 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962141 5047 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962143 5047 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962146 5047 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962149 5047 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962152 5047 flags.go:64] FLAG: --lock-file="" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962155 5047 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962158 5047 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962161 5047 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962167 5047 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962170 5047 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962173 5047 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962178 5047 flags.go:64] FLAG: --logging-format="text" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962181 5047 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962185 5047 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962188 5047 flags.go:64] FLAG: --manifest-url="" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962191 5047 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962197 5047 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962200 5047 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962204 5047 flags.go:64] FLAG: --max-pods="110" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962208 5047 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962211 5047 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962214 5047 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:04:06.975939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962217 5047 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962220 5047 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962224 5047 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962227 5047 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962234 5047 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962238 5047 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962241 5047 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962244 5047 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962247 5047 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962250 5047 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962253 5047 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962256 5047 flags.go:64] FLAG: --port="10250" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962260 5047 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962263 5047 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02cb9b13e1dc18340" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962266 5047 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962269 5047 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962273 5047 flags.go:64] FLAG: --register-node="true" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962276 5047 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962280 5047 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962283 5047 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962286 5047 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962290 5047 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962293 5047 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962297 5047 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962300 5047 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962303 5047 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:04:06.977087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962306 5047 flags.go:64] FLAG: --runonce="false" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962309 5047 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962312 5047 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962315 5047 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962319 5047 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962322 5047 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962325 5047 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962328 5047 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962332 5047 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962335 5047 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962338 5047 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962341 5047 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962344 5047 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962348 5047 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962351 5047 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962354 5047 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962360 5047 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962363 5047 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962366 5047 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962371 5047 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962374 5047 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962377 5047 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962380 5047 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962383 5047 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962386 5047 flags.go:64] FLAG: --v="2" Oct 30 19:04:06.977858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962391 5047 flags.go:64] FLAG: --version="false" Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962395 5047 flags.go:64] FLAG: --vmodule="" Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962401 5047 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962404 5047 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962535 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962539 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962542 5047 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962545 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962548 5047 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962551 5047 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962553 5047 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962556 5047 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962559 5047 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962562 5047 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962565 5047 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962568 5047 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962571 5047 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962575 5047 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962579 5047 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:04:06.978494 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962582 5047 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962585 5047 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962588 5047 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962591 5047 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962594 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962597 5047 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962599 5047 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962602 5047 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962605 5047 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962607 5047 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962610 5047 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962612 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962615 5047 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962618 5047 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962621 5047 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962625 5047 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962628 5047 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962631 5047 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962633 5047 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962636 5047 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:04:06.978976 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962639 5047 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962641 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962644 5047 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962647 5047 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962650 5047 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962652 5047 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962656 5047 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962658 5047 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962661 5047 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962664 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962666 5047 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962669 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962672 5047 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962675 5047 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962677 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962680 5047 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962682 5047 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962685 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962687 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962690 5047 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:04:06.979484 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962693 5047 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962696 5047 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962698 5047 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962701 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962704 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962708 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962710 5047 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962714 5047 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962717 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962720 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962722 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962725 5047 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962728 5047 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962730 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962733 5047 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962736 5047 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962738 5047 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962741 5047 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962745 5047 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:04:06.979964 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962750 5047 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962753 5047 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962755 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962758 5047 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962761 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962764 5047 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962766 5047 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962769 5047 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962771 5047 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962774 5047 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962777 5047 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962779 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962782 5047 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962785 5047 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962787 5047 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962790 5047 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962793 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962796 5047 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962800 5047 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:04:06.980586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962802 5047 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:04:06.981054 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962806 5047 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:04:06.981054 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962809 5047 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:04:06.981054 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962812 5047 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:04:06.981054 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962814 5047 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:04:06.981054 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962817 5047 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:04:06.981054 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962819 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:04:06.981054 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962822 5047 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:04:06.981054 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962825 5047 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:04:06.981054 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962828 5047 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:04:06.981054 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962830 5047 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:04:06.981054 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962833 5047 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:04:06.981054 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962835 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:04:06.981054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.962862 5047 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:04:06.981054 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962901 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:04:06.981054 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962906 5047 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:04:06.981054 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962909 5047 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962912 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962915 5047 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962918 5047 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962920 5047 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962923 5047 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962927 5047 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962931 5047 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962934 5047 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962936 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962939 5047 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962942 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962945 5047 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962947 5047 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962950 5047 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962954 5047 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962957 5047 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962960 5047 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962963 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:04:06.981476 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962966 5047 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962969 5047 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962972 5047 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962974 5047 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962977 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962980 5047 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962983 5047 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962985 5047 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962988 5047 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962991 5047 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962994 5047 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.962997 5047 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963000 5047 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963016 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963019 5047 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963022 5047 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963025 5047 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963028 5047 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963031 5047 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:04:06.981973 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963034 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963037 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963040 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963043 5047 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963045 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963048 5047 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963051 5047 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963053 5047 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963056 5047 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963060 5047 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963063 5047 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963066 5047 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963069 5047 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963072 5047 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963076 5047 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963080 5047 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963083 5047 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963086 5047 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963089 5047 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:04:06.982465 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963092 5047 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963094 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963097 5047 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963099 5047 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963102 5047 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963105 5047 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963107 5047 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963110 5047 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963113 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963117 5047 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963119 5047 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963122 5047 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963125 5047 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963127 5047 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963130 5047 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963132 5047 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963135 5047 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963138 5047 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963141 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963143 5047 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:04:06.982967 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963146 5047 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963149 5047 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963153 5047 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963155 5047 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963158 5047 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963161 5047 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963164 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963167 5047 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963169 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963172 5047 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963174 5047 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963177 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963180 5047 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963182 5047 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963185 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963188 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963190 5047 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963193 5047 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963196 5047 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963199 5047 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:04:06.983960 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963202 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963205 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963208 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963210 5047 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963213 5047 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963216 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.963218 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.966916 5047 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.966933 5047 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.966951 5047 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.966956 5047 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967149 5047 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967531 5047 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967538 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967545 5047 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967551 5047 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967556 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967561 5047 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967574 5047 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967579 5047 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:04:06.984749 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967585 5047 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967590 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967594 5047 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967599 5047 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967604 5047 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967609 5047 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967613 5047 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967618 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967623 5047 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967628 5047 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967633 5047 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967643 5047 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967648 5047 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967652 5047 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967657 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967661 5047 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967677 5047 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967682 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967686 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967691 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:04:06.985324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967695 5047 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967702 5047 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967709 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967721 5047 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967725 5047 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967731 5047 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967736 5047 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967740 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967745 5047 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967749 5047 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967754 5047 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967758 5047 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967766 5047 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967770 5047 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967775 5047 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967779 5047 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967789 5047 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967793 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967797 5047 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:04:06.985818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967801 5047 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967806 5047 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967810 5047 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967815 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967819 5047 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967824 5047 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967828 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967833 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967837 5047 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967842 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967852 5047 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967856 5047 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967861 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967865 5047 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967869 5047 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967874 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967878 5047 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967883 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967888 5047 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:04:06.986335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967892 5047 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967897 5047 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967901 5047 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967911 5047 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967916 5047 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967920 5047 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967926 5047 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967930 5047 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967934 5047 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967939 5047 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967943 5047 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967948 5047 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967953 5047 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967958 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967962 5047 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967967 5047 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967976 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967981 5047 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967985 5047 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967990 5047 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:04:06.986795 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967994 5047 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.967998 5047 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968025 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968033 5047 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968041 5047 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968046 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968050 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968055 5047 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968064 5047 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968069 5047 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968074 5047 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968078 5047 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968083 5047 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968087 5047 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968091 5047 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968096 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968100 5047 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968105 5047 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:04:06.987545 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968111 5047 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:04:06.988313 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.968173 5047 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:04:06.988313 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968599 5047 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:04:06.988313 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968610 5047 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:04:06.988313 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968615 5047 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:04:06.988313 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968618 5047 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:04:06.988313 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968622 5047 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:04:06.988313 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968625 5047 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:04:06.988313 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968630 5047 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:04:06.988313 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968632 5047 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:04:06.988313 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968636 5047 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:04:06.988313 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968638 5047 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:04:06.988313 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968641 5047 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:04:06.988313 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968644 5047 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:04:06.988313 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968648 5047 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:04:06.988313 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968651 5047 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:04:06.988313 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968654 5047 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:04:06.988313 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968656 5047 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968659 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968662 5047 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968665 5047 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968668 5047 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968671 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968674 5047 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968677 5047 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968679 5047 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968682 5047 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968685 5047 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968687 5047 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968690 5047 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968693 5047 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968696 5047 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968699 5047 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968701 5047 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968704 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968707 5047 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:04:06.988985 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968710 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968712 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968715 5047 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968717 5047 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968721 5047 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968724 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968727 5047 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968729 5047 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968732 5047 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968735 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968737 5047 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968740 5047 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968744 5047 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968749 5047 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968753 5047 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968755 5047 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968758 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968761 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968764 5047 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:04:06.989800 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968766 5047 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968769 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968772 5047 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968775 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968780 5047 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968783 5047 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968786 5047 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968788 5047 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968791 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968794 5047 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968797 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968800 5047 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968802 5047 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968805 5047 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968808 5047 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968811 5047 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968814 5047 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968817 5047 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968820 5047 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968823 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:04:06.990347 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968826 5047 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968828 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968832 5047 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968835 5047 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968838 5047 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968841 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968844 5047 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968846 5047 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968849 5047 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968852 5047 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968855 5047 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968857 5047 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968860 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968863 5047 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968865 5047 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968868 5047 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968871 5047 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968874 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968877 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:04:06.990865 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968880 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:04:06.991357 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968883 5047 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:04:06.991357 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968885 5047 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:04:06.991357 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968888 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:04:06.991357 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968891 5047 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:04:06.991357 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968895 5047 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:04:06.991357 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968899 5047 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:04:06.991357 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968902 5047 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:04:06.991357 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968905 5047 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:04:06.991357 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968908 5047 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:04:06.991357 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968912 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:04:06.991357 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968914 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:04:06.991357 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968917 5047 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:04:06.991357 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968920 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:04:06.991357 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.968949 5047 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:04:06.991357 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968985 5047 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968990 5047 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968993 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968996 5047 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.968999 5047 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969022 5047 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969027 5047 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969031 5047 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969035 5047 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969039 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969043 5047 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969047 5047 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969051 5047 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969055 5047 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969059 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969065 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969070 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969074 5047 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969078 5047 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:04:06.991783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969082 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969086 5047 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969091 5047 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969096 5047 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969100 5047 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969102 5047 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969106 5047 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969109 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969113 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969116 5047 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969119 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969121 5047 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969124 5047 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969127 5047 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969129 5047 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969134 5047 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969137 5047 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969140 5047 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969143 5047 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:04:06.992379 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969145 5047 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969148 5047 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969151 5047 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969153 5047 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969156 5047 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969159 5047 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969162 5047 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969164 5047 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969167 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969171 5047 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969173 5047 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969176 5047 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969180 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969182 5047 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969185 5047 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969188 5047 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969190 5047 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969193 5047 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969196 5047 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969198 5047 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:04:06.992899 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969201 5047 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969206 5047 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969208 5047 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969211 5047 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969214 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969216 5047 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969219 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969222 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969224 5047 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969228 5047 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969231 5047 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969234 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969236 5047 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969239 5047 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969241 5047 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969244 5047 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969246 5047 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969249 5047 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969253 5047 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:04:06.993456 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969257 5047 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969260 5047 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969264 5047 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969267 5047 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969270 5047 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969272 5047 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969275 5047 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969278 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969281 5047 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969284 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969286 5047 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969289 5047 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969292 5047 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969294 5047 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969299 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969302 5047 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969305 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969307 5047 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969310 5047 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969313 5047 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:04:06.993925 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969315 5047 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:04:06.994436 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969318 5047 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:04:06.994436 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969321 5047 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:04:06.994436 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969324 5047 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:04:06.994436 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969326 5047 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:04:06.994436 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969329 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:04:06.994436 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969331 5047 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:04:06.994436 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969334 5047 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:04:06.994436 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:06.969337 5047 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:04:06.994436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.969401 5047 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:04:06.994436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.970416 5047 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 19:04:06.994436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.970473 5047 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 19:04:06.994436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.970644 5047 server.go:1014] "Starting client certificate rotation" Oct 30 19:04:06.994436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.970783 5047 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:04:06.994436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.970842 5047 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:58:26 +0000 UTC" deadline="2028-05-03 01:48:59.820041308 +0000 UTC" Oct 30 19:04:06.994436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.970873 5047 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13206h44m52.849171965s" Oct 30 19:04:06.994436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.971099 5047 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:04:06.994841 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.972167 5047 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:04:06.994841 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.973311 5047 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:04:06.994841 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.974969 5047 log.go:25] "Validated CRI v1 image API" Oct 30 19:04:06.994841 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.975750 5047 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:04:06.994841 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.977768 5047 fs.go:135] Filesystem UUIDs: map[4ea0243f-3872-4281-9afc-62a44536e3b6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a7a49493-b336-4bb8-8627-6a373f7aac9e:/dev/nvme0n1p4] Oct 30 19:04:06.995068 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.977791 5047 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1107f3e4351949dcfe0a9516bdcf58cd346932add8fa00e07c869b68efcea9f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1107f3e4351949dcfe0a9516bdcf58cd346932add8fa00e07c869b68efcea9f1/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11beabe9c6de3cd56340ccae3df3b8dae40820c3bc49d480260941265241cbdd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11beabe9c6de3cd56340ccae3df3b8dae40820c3bc49d480260941265241cbdd/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ab31aab904cf00ea369247612bac2a6903a09991025c519e691a647ecab581f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ab31aab904cf00ea369247612bac2a6903a09991025c519e691a647ecab581f/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/235840f48c7642fabfb8577e4dc8aae46f92e2b241626f9eda4b30ae41a26152/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/235840f48c7642fabfb8577e4dc8aae46f92e2b241626f9eda4b30ae41a26152/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/287bd230c933af1b79430cbd50c3f852bdf540f15b5b12a8df5b7853869af3fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/287bd230c933af1b79430cbd50c3f852bdf540f15b5b12a8df5b7853869af3fc/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/376eabfad4be616716b578f3918e6800b8fe2eb6a7ce7b84cb32e95810229ef3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/376eabfad4be616716b578f3918e6800b8fe2eb6a7ce7b84cb32e95810229ef3/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/400470efa985001a73273717eec4096cb361f495ba0b0e9aeaba126ebb524a63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/400470efa985001a73273717eec4096cb361f495ba0b0e9aeaba126ebb524a63/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b378dcf907d51ccab7982b3c15d671b97fae5af79e334b69b5b0594b5f61c0a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b378dcf907d51ccab7982b3c15d671b97fae5af79e334b69b5b0594b5f61c0a/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69b1e146aaabdee04685a2656258c11196f9e39a0605b2629b4fd0732be785c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69b1e146aaabdee04685a2656258c11196f9e39a0605b2629b4fd0732be785c3/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70748cb186264e6b7d4b61ec14812cd80f483a9b29732084126e01652d7db72e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70748cb186264e6b7d4b61ec14812cd80f483a9b29732084126e01652d7db72e/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70a7967543e8a1c5cd506bff801efa08d3a298510b88e1a03def33cee2a138d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70a7967543e8a1c5cd506bff801efa08d3a298510b88e1a03def33cee2a138d1/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74050fbb7ca681a4b0d0176a5d938d780a131034dfe048fef03ee3dcbc83aded/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74050fbb7ca681a4b0d0176a5d938d780a131034dfe048fef03ee3dcbc83aded/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a6e72a2cf13bb7fba89ef5293f61f7f1ee7ab4085817a8f80353dd02bae8b8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a6e72a2cf13bb7fba89ef5293f61f7f1ee7ab4085817a8f80353dd02bae8b8a/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7fe8396ac1bb3f74a1e74c4c62f25b12e0eb09e446a3712167e48e0e6154dec4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7fe8396ac1bb3f74a1e74c4c62f25b12e0eb09e446a3712167e48e0e6154dec4/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91fc483e041346371d853e49b581b082802ddff668b1d2893b35150ba16c0aa2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91fc483e041346371d853e49b581b082802ddff668b1d2893b35150ba16c0aa2/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5e0b634a45229b1af26839f8414ee1ac9f2c372dd0acb5e8862e567bf6889d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5e0b634a45229b1af26839f8414ee1ac9f2c372dd0acb5e8862e567bf6889d4/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aaa13a46135b4def3571bba4900cdd96c05d508a0aea8d03d55b21e79f216836/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aaa13a46135b4def3571bba4900cdd96c05d508a0aea8d03d55b21e79f216836/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb8e5379f32304111cf0192bb8be7c0f2484ec432c185480ca4af01bb1f32133/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb8e5379f32304111cf0192bb8be7c0f2484ec432c185480ca4af01bb1f32133/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bcb44cd7dc162cffc616356e2a7bd5e54c05396519e79f2008d54d2a075f51ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bcb44cd7dc162cffc616356e2a7bd5e54c05396519e79f2008d54d2a075f51ad/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df0274035fb0fb4f6a09a0d067b69f2f73173263d4f4e814bd26e550a97f566e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df0274035fb0fb4f6a09a0d067b69f2f73173263d4f4e814bd26e550a97f566e/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e118e1579fbf034057e029e9bf4c55fc16e8477097cb06e1ed0245d2be15f79d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e118e1579fbf034057e029e9bf4c55fc16e8477097cb06e1ed0245d2be15f79d/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5d5df8b1241cedec650cf6a47103044b6c2ea856b1f043f15fdd81ec60666f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5d5df8b1241cedec650cf6a47103044b6c2ea856b1f043f15fdd81ec60666f0/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9365e876ff21c8086855f6e1563d090b4aebb43c65f350da97ddaf74a361471/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9365e876ff21c8086855f6e1563d090b4aebb43c65f350da97ddaf74a361471/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea757d4eaa177307762b1f6748e8cb48eccf56fd484c16a3bd72ed4378137412/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea757d4eaa177307762b1f6748e8cb48eccf56fd484c16a3bd72ed4378137412/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f641733acf22c32e20702d19d3e0226b46203b223b7985d7ca60947e7541184f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f641733acf22c32e20702d19d3e0226b46203b223b7985d7ca60947e7541184f/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fad4b0bf404d53e190cb2aa2851957981a46518e308a53dc7a31ef760e520314/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fad4b0bf404d53e190cb2aa2851957981a46518e308a53dc7a31ef760e520314/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11d31a53-ae1b-4d46-aa8c-6c00e89ed539/volumes/kubernetes.io~projected/kube-api-access-77qjb:{mountpoint:/var/lib/kubelet/pods/11d31a53-ae1b-4d46-aa8c-6c00e89ed539/volumes/kubernetes.io~projected/kube-api-access-77qjb major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11d31a53-ae1b-4d46-aa8c-6c00e89ed539/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/11d31a53-ae1b-4d46-aa8c-6c00e89ed539/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a7defa5-7e66-4e17-8dc8-e74137e4f054/volumes/kubernetes.io~projected/kube-api-access-gmxmw:{mountpoint:/var/lib/kubelet/pods/1a7defa5-7e66-4e17-8dc8-e74137e4f054/volumes/kubernetes.io~projected/kube-api-access-gmxmw major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20403b1e-e479-45b9-bb85-52cf10725d86/volumes/kubernetes.io~projected/kube-api-access-6rxlq:{mountpoint:/var/lib/kubelet/pods/20403b1e-e479-45b9-bb85-52cf10725d86/volumes/kubernetes.io~projected/kube-api-access-6rxlq major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20403b1e-e479-45b9-bb85-52cf10725d86/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/20403b1e-e479-45b9-bb85-52cf10725d86/volumes/kubernetes.io~secret/default-certificate major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20403b1e-e479-45b9-bb85-52cf10725d86/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/20403b1e-e479-45b9-bb85-52cf10725d86/volumes/kubernetes.io~secret/stats-auth major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28f40fee-93d5-47de-89aa-943fa6b4efe7/volumes/kubernetes.io~projected/kube-api-access-q9r9b:{mountpoint:/var/lib/kubelet/pods/28f40fee-93d5-47de-89aa-943fa6b4efe7/volumes/kubernetes.io~projected/kube-api-access-q9r9b major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ac3ab69-eead-4eca-af11-53fec376a6ac/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/3ac3ab69-eead-4eca-af11-53fec376a6ac/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ac3ab69-eead-4eca-af11-53fec376a6ac/volumes/kubernetes.io~projected/kube-api-access-zn7mf:{mountpoint:/var/lib/kubelet/pods/3ac3ab69-eead-4eca-af11-53fec376a6ac/volumes/kubernetes.io~projected/kube-api-access-zn7mf major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ac3ab69-eead-4eca-af11-53fec376a6ac/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/3ac3ab69-eead-4eca-af11-53fec376a6ac/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ac3ab69-eead-4eca-af11-53fec376a6ac/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/3ac3ab69-eead-4eca-af11-53fec376a6ac/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e626b03-28b0-4840-bbe2-518834311815/volumes/kubernetes.io~projected/kube-api-access-wtd7p:{mountpoint:/var/lib/kubelet/pods/3e626b03-28b0-4840-bbe2-518834311815/volumes/kubernetes.io~projected/kube-api-access-wtd7p major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e626b03-28b0-4840-bbe2-518834311815/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3e626b03-28b0-4840-bbe2-518834311815/volumes/kubernetes.io~secret/cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/415a1d40-3275-4624-98cd-3df86bbc819a/volumes/kubernetes.io~projected/kube-api-access-slnzw:{mountpoint:/var/lib/kubelet/pods/415a1d40-3275-4624-98cd-3df86bbc819a/volumes/kubernetes.io~projected/kube-api-access-slnzw major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/452b3edc-e035-446c-9b7b-ee8154f3c6c3/volumes/kubernetes.io~projected/kube-api-access-vlw9l:{mountpoint:/var/lib/kubelet/pods/452b3edc-e035-446c-9b7b-ee8154f3c6c3/volumes/kubernetes.io~projected/kube-api-access-vlw9l major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/540944ec-348d-46e1-836b-9aeb07d16713/volumes/kubernetes.io~projected/kube-api-access-6dbws:{mountpoint:/var/lib/kubelet/pods/540944ec-348d-46e1-836b-9aeb07d16713/volumes/kubernetes.io~projected/kube-api-access-6dbws major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5557e025-30b7-4315-b7a1-308e166c9a3b/volumes/kubernetes.io~projected/kube-api-access-j4jzx:{mountpoint:/var/lib/kubelet/pods/5557e025-30b7-4315-b7a1-308e166c9a3b/volumes/kubernetes.io~projected/kube-api-access-j4jzx major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57ee29d6-ad4d-4ed8-ae1a-877732a3737b/volumes/kubernetes.io~projected/kube-api-access-tvqk8:{mountpoint:/var/lib/kubelet/pods/57ee29d6-ad4d-4ed8-ae1a-877732a3737b/volumes/kubernetes.io~projected/kube-api-access-tvqk8 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/633f6b63-a067-470b-bec9-e913e7f28589/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/633f6b63-a067-470b-bec9-e913e7f28589/volumes/kubernetes.io~secret/agent-certs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68061b2f-50b6-456d-86e7-4b766c7925bc/volumes/kubernetes.io~projected/kube-api-access-qlsmw:{mountpoint:/var/lib/kubelet/pods/68061b2f-50b6-456d-86e7-4b766c7925bc/volumes/kubernetes.io~projected/kube-api-access-qlsmw major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68061b2f-50b6-456d-86e7-4b766c7925bc/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/68061b2f-50b6-456d-86e7-4b766c7925bc/volumes/kubernetes.io~secret/signing-key major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6af08cb7-fa60-491b-88fd-07c1ace45779/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/6af08cb7-fa60-491b-88fd-07c1ace45779/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6af08cb7-fa60-491b-88fd-07c1ace45779/volumes/kubernetes.io~projected/kube-api-access-sp2w6:{mountpoint:/var/lib/kubelet/pods/6af08cb7-fa60-491b-88fd-07c1ace45779/volumes/kubernetes.io~projected/kube-api-access-sp2w6 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6af08cb7-fa60-491b-88fd-07c1ace45779/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/6af08cb7-fa60-491b-88fd-07c1ace45779/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6af08cb7-fa60-491b-88fd-07c1ace45779/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/6af08cb7-fa60-491b-88fd-07c1ace45779/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/721cf657-76b1-4fdf-868d-d84cb9fa0b14/volumes/kubernetes.io~projected/kube-api-access-bphkr:{mountpoint:/var/lib/kubelet/pods/721cf657-76b1-4fdf-868d-d84cb9fa0b14/volumes/kubernetes.io~projected/kube-api-access-bphkr major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/721cf657-76b1-4fdf-868d-d84cb9fa0b14/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/721cf657-76b1-4fdf-868d-d84cb9fa0b14/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad/volumes/kubernetes.io~projected/kube-api-access-p9gkl:{mountpoint:/var/lib/kubelet/pods/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad/volumes/kubernetes.io~projected/kube-api-access-p9gkl major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/830ddb4e-a8f8-406c-9e89-755bb0d48e0e/volumes/kubernetes.io~projected/kube-api-access-stb92:{mountpoint:/var/lib/kubelet/pods/830ddb4e-a8f8-406c-9e89-755bb0d48e0e/volumes/kubernetes.io~projected/kube-api-access-stb92 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ae35879-1912-43b0-a7a5-9ac064ce0dea/volumes/kubernetes.io~projected/kube-api-access-zhzdr:{mountpoint:/var/lib/kubelet/pods/8ae35879-1912-43b0-a7a5-9ac064ce0dea/volumes/kubernetes.io~projected/kube-api-access-zhzdr major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93f0a70e-357e-4d65-b554-42363280bebd/volumes/kubernetes.io~projected/kube-api-access-m4mmt:{mountpoint:/var/lib/kubelet/pods/93f0a70e-357e-4d65-b554-42363280bebd/volumes/kubernetes.io~projected/kube-api-access-m4mmt major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/969393ae-3d0e-4f83-bf03-0396c386ffbf/volumes/kubernetes.io~projected/kube-api-access-8dfz6:{mountpoint:/var/lib/kubelet/pods/969393ae-3d0e-4f83-bf03-0396c386ffbf/volumes/kubernetes.io~projected/kube-api-access-8dfz6 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b/volumes/kubernetes.io~projected/kube-api-access-bdw7x:{mountpoint:/var/lib/kubelet/pods/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b/volumes/kubernetes.io~projected/kube-api-access-bdw7x major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8d10e22-9215-4fc2-8474-5bc68cb57f18/volumes/kubernetes.io~projected/kube-api-access-bfkbg:{mountpoint:/var/lib/kubelet/pods/a8d10e22-9215-4fc2-8474-5bc68cb57f18/volumes/kubernetes.io~projected/kube-api-access-bfkbg major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad4cd78b-b491-4927-80ae-dae026d10e7a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ad4cd78b-b491-4927-80ae-dae026d10e7a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b50175d2-e6bb-463a-9602-5bf9e4719a15/volumes/kubernetes.io~projected/kube-api-access-2d65z:{mountpoint:/var/lib/kubelet/pods/b50175d2-e6bb-463a-9602-5bf9e4719a15/volumes/kubernetes.io~projected/kube-api-access-2d65z major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd7cda5e-1a4b-45f8-9cab-f96111986c08/volumes/kubernetes.io~projected/kube-api-access-w9mcb:{mountpoint:/var/lib/kubelet/pods/bd7cda5e-1a4b-45f8-9cab-f96111986c08/volumes/kubernetes.io~projected/kube-api-access-w9mcb major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd7cda5e-1a4b-45f8-9cab-f96111986c08/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/bd7cda5e-1a4b-45f8-9cab-f96111986c08/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfcde492-9464-42cc-a01c-84d9eec4135a/volumes/kubernetes.io~projected/kube-api-access-9kd76:{mountpoint:/var/lib/kubelet/pods/bfcde492-9464-42cc-a01c-84d9eec4135a/volumes/kubernetes.io~projected/kube-api-access-9kd76 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6188415-c8d5-4972-9d6d-f6e3bcf4b079/volumes/kubernetes.io~projected/kube-api-access-8x9z4:{mountpoint:/var/lib/kubelet/pods/c6188415-c8d5-4972-9d6d-f6e3bcf4b079/volumes/kubernetes.io~projected/kube-api-access-8x9z4 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6188415-c8d5-4972-9d6d-f6e3bcf4b079/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c6188415-c8d5-4972-9d6d-f6e3bcf4b079/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caef3ef2-f9af-4421-866e-61cc4febba1a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/caef3ef2-f9af-4421-866e-61cc4febba1a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caef3ef2-f9af-4421-866e-61cc4febba1a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/caef3ef2-f9af-4421-866e-61cc4febba1a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caef3ef2-f9af-4421-866e-61cc4febba1a/volumes/kubernetes.io~projected/kube-api-access-58fsk:{mountpoint:/var/lib/kubelet/pods/caef3ef2-f9af-4421-866e-61cc4febba1a/volumes/kubernetes.io~projected/kube-api-access-58fsk major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ceff794b-7638-49c7-a0f4-495a8cf3d8c7/volumes/kubernetes.io~projected/kube-api-access-qm9bn:{mountpoint:/var/lib/kubelet/pods/ceff794b-7638-49c7-a0f4-495a8cf3d8c7/volumes/kubernetes.io~projected/kube-api-access-qm9bn major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3c525ed-5017-482f-ac5c-670002363472/volumes/kubernetes.io~projected/kube-api-access-brtql:{mountpoint:/var/lib/kubelet/pods/d3c525ed-5017-482f-ac5c-670002363472/volumes/kubernetes.io~projected/kube-api-access-brtql major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee4d07b7-2acc-456e-932b-bfefb2f666df/volumes/kubernetes.io~projected/kube-api-access-ftrxn:{mountpoint:/var/lib/kubelet/pods/ee4d07b7-2acc-456e-932b-bfefb2f666df/volumes/kubernetes.io~projected/kube-api-access-ftrxn major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7537906-9d0e-4b31-9d76-45936780d55c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f7537906-9d0e-4b31-9d76-45936780d55c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7537906-9d0e-4b31-9d76-45936780d55c/volumes/kubernetes.io~projected/kube-api-access-r5xn9:{mountpoint:/var/lib/kubelet/pods/f7537906-9d0e-4b31-9d76-45936780d55c/volumes/kubernetes.io~projected/kube-api-access-r5xn9 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7537906-9d0e-4b31-9d76-45936780d55c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f7537906-9d0e-4b31-9d76-45936780d55c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/facfc926-9288-41c4-bdc3-5d8acf446292/volumes/kubernetes.io~projected/kube-api-access-m72r5:{mountpoint:/var/lib/kubelet/pods/facfc926-9288-41c4-bdc3-5d8acf446292/volumes/kubernetes.io~projected/kube-api-access-m72r5 major:0 minor:340 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/96cbbb7613391761198dd513612e564dd2a8e22f5c69d08935c242385cf2d554/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a8e2f7e13570d6adb7999356df99aff4ec1b9ed3163c1de5d2924307eeca1157/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1c0fe1cc0debf79e291da90aca26c951dc68a247edb65a0b49ced6f0c5cd0ab6/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/43ec95b24e778c6e767e2cc9a31f4d0b94a0bb20bbb0e1ee530897b67009fc02/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/fcbbf10aa8dc6bcb61b13e009a8f86bd839319cf76e8507c1d071b9c8e40a536/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ee4b6322ee348f44d28b2a49d399894ab4c9555d6aff1d996122e2093d26f2c4/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/b31b623d1ac4dfcef4a09262d4110afecb9664be029a9d3e7d2463c4d219bbe0/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/48e6493152d3990938a7f4f6e6c1014e0d7a356b729369f67aad7f560850a8e9/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/236c6be969e03ccb7410ead7b702f3096c00645dbdc7e3cf551a1aa985531ede/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/c16bd4d9c45fc9fb253e09716ea879f240efc394707b28c1402533a257eec47e/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/5436d2cfd78d7b0db375c52e845b848a885e59e155723aacd847b2ceac3a91e5/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/a273e38d1b39452de980fd9386b6a6ec0c1a14d99345decc79327e3e161c32d4/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/07e96e85464152f3ddc62a269a39c3e3a5aaa7283dc45572300f0c47ad793a42/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1c847cc9c4e1bb257124e3c655c6c82297c021ec5287471dac63b4ead9f38d21/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/61e4755f31efadf6185ef424aa7c6b0d8824a478f4c9f69199a73aea424432bb/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/b19170fe448a7095c0b3fa9cd57e03bb9f8d070883694c45d9fb777579d4682e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b54662edeeb44b1ae413aab121a5b517c45cf7fdb4236eb6c3788cb11fab3a70/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9b968d255703632076d1fc78714e0482f3bb6a47712c3dc1b4ff541f24296f4d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c422c6202ca7548dbd5d3031cc725a0192c51c7dea65b07c4b8c6a0e7f625117/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/7ae119f1539dac4f290dd4ad2db18fd7b9587434642489bf4bab6da16f75333e/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/a9e12c36c57c34eb432d595eabcd05e109312ddbd12b29bbcd86fbb473f807bf/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/5beb0b7a11bbfce5b3dfe24906091827a2c9aeb03407e33a8abeef25351f304f/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/619ee5a6f86d3ba930bd3f39bbd91962bcd8c15d19ea5ee38fe3f254d66eb5f5/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/e45df073d7c1d33562a33363af744d7ee862eac2f01401d781bd2c4d93216ff6/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/0cf6d20ff33021662208be20bad7bd1aa4698e67ab7f1b9dd9e62c0f31cfd0a3/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/e4ab67a16d71d1da4097c3cba1202023de3b3f1f457ea3fdce7ceb88f2182829/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/299ba9f5030e61472bea8e381542ae7721f389e37d0a42df559ee9e33aade1cb/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/8145862d0806038f3b2dfc8f94195735b14a94149c484e13b95d7c3ee56cea84/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/4237965d172e3126b79d8208b093ce6fc972a6a032fcf39a8b20cb7fb7b0ba28/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/5babb50cf4a3038809a00f94fb26f9b71c24d72dae5e8ac7942f9226fd1677bd/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/88ac620456e2ae25be92472ff2b669315c601342b223a404be40c694b2adab16/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/5b7cc7afc0a42333c4ed1f91701b83531dccfc5e63d2e8daa34c6272160068ab/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/da335521f571892177857d6fe6f9d29a9d58c048eccdcadae139d588641cc4ea/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/f514106e66bdca5b24d5068253cc6c95ed18206ffc7207bd09fd9c8cb24d86c5/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/43c31087947cc026222389ba53f911156bd1b7c7e57dab5fd3d5b6c1b12a7bff/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/15d0956ed85b56e1ab917b36fd763eeefdd5108724bf12c88d29987628af41b6/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/d7d8a502b6e6fd49438b46e511e4b24569b0bc6f13f0fafc2952a199bef80ba9/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/239d5d2877e2cc397d3371eb218c487b362139fd559d30e1b464182ca67a4f40/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/5aa48d73bf97be54a0881dce3787ccc7c929fef0d955e781dca470c53c3cdb44/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/3718f1348fc1ac055583380010c60c1a1d7f9e92af0346587ec7b5182127f4bf/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/d680f99e913cc098324132cad285ac6f618d2d389556b8dfceb413b956dbad4e/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/07dde00f33f26b6c7e54501117884492caf7a8215e722ac0776adcadc1b23dc4/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/9a45d2caba00f9ee7dece58b9656441df1385017a0a493efc36b1174c5a079e9/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/63dede9af10c614728e13a247845bbbf61dec082d1c02b1951b09e3139e16280/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/b1eb6329fff78e1526bd6cd2f895b94fab5dd98e44bcd209a32fb4982321f011/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/ee6dd6b830a3877d9df1f60abe475963a2619bdfe88eb04fa687e4f8c6582b43/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/a869b2929c559da57ca13cf2bfd0f4fa2ecda873d5486285c831d21fce257b26/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/4fa9d5f17af2ca3b184e6aa7faf88bcea94455896fb9c730a67de1c844ad4c06/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/aaab08b42211c2089930de0961cd27259d997d5ab95a3380f4b4342680f312fa/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/a02de4e2078f2b54b281adb7937bbd19d79604e7981070acaac70d992f09fe41/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/0658a2df5451633ab6c3b52b571ee3b6d9aaeb7648dbf72f8c4d743ad07d6731/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/ff8f7df6397c3fa5f3057bb64f62666fb2be6c3daabde5e49751052578b307d5/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/b8a94ddc6f0b66bb8b53fa0b5f2afd8624fb94e3657f9867ffa804f6b0159833/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/2135e82f0552ecead3b42dc7e0794ad0a4aad974052b94897987a95cc6705237/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/4f2c7f6703ffd0cedc71a96d109e3b14184d7504db3845692d2228d9b9deccda/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/84a76f0a0649391c53e1271f3fc2e2231bf9740ca4e8269d26a54603b461a4ca/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/cf4c0979a02916c49a7784af5576fe0443ccbe22818fa462e2a5a71861f716b1/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/3616f0680e5570ba45d4db89fd331d10447dd1c1575d63703144469aa4e541c1/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/6c4a0045806048f494397508d0262b697cb52913d5fdc0d4dc46b2f4c4363da5/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/6de073716df36fe35c75a48b79270afccf31aa9dd438f4efab361f577d4b165a/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/1bc9ce9a7dfaaa6a43054674317df2700e96ef546aa0c10b4a306a535381b24f/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/b30fae20d1904d3e6ccd01d1251f6bc7e50afedcaab6ffe55b40450888d050ee/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.986557 5047 manager.go:217] Machine: {Timestamp:2026-10-30 19:04:06.986097083 +0000 UTC m=+0.060552433 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095510 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ad70a9a7e50c62fc39b52d44113a5 SystemUUID:ec2ad70a-9a7e-50c6-2fc3-9b52d44113a5 BootID:4f9335b3-918d-4333-a3f7-cd383854f7ad Filesystems:[{Device:/run/containers/storage/overlay-containers/ea757d4eaa177307762b1f6748e8cb48eccf56fd484c16a3bd72ed4378137412/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b378dcf907d51ccab7982b3c15d671b97fae5af79e334b69b5b0594b5f61c0a/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/facfc926-9288-41c4-bdc3-5d8acf446292/volumes/kubernetes.io~projected/kube-api-access-m72r5 DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b50175d2-e6bb-463a-9602-5bf9e4719a15/volumes/kubernetes.io~projected/kube-api-access-2d65z DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3e626b03-28b0-4840-bbe2-518834311815/volumes/kubernetes.io~projected/kube-api-access-wtd7p DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e626b03-28b0-4840-bbe2-518834311815/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fad4b0bf404d53e190cb2aa2851957981a46518e308a53dc7a31ef760e520314/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69b1e146aaabdee04685a2656258c11196f9e39a0605b2629b4fd0732be785c3/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3ac3ab69-eead-4eca-af11-53fec376a6ac/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1107f3e4351949dcfe0a9516bdcf58cd346932add8fa00e07c869b68efcea9f1/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b/volumes/kubernetes.io~projected/kube-api-access-bdw7x DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb8e5379f32304111cf0192bb8be7c0f2484ec432c185480ca4af01bb1f32133/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/415a1d40-3275-4624-98cd-3df86bbc819a/volumes/kubernetes.io~projected/kube-api-access-slnzw DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd7cda5e-1a4b-45f8-9cab-f96111986c08/volumes/kubernetes.io~projected/kube-api-access-w9mcb DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aaa13a46135b4def3571bba4900cdd96c05d508a0aea8d03d55b21e79f216836/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e118e1579fbf034057e029e9bf4c55fc16e8477097cb06e1ed0245d2be15f79d/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ac3ab69-eead-4eca-af11-53fec376a6ac/volumes/kubernetes.io~projected/kube-api-access-zn7mf DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74050fbb7ca681a4b0d0176a5d938d780a131034dfe048fef03ee3dcbc83aded/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5d5df8b1241cedec650cf6a47103044b6c2ea856b1f043f15fdd81ec60666f0/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bcb44cd7dc162cffc616356e2a7bd5e54c05396519e79f2008d54d2a075f51ad/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3ac3ab69-eead-4eca-af11-53fec376a6ac/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6af08cb7-fa60-491b-88fd-07c1ace45779/volumes/kubernetes.io~projected/kube-api-access-sp2w6 DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/540944ec-348d-46e1-836b-9aeb07d16713/volumes/kubernetes.io~projected/kube-api-access-6dbws DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a6e72a2cf13bb7fba89ef5293f61f7f1ee7ab4085817a8f80353dd02bae8b8a/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df0274035fb0fb4f6a09a0d067b69f2f73173263d4f4e814bd26e550a97f566e/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfcde492-9464-42cc-a01c-84d9eec4135a/volumes/kubernetes.io~projected/kube-api-access-9kd76 DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/20403b1e-e479-45b9-bb85-52cf10725d86/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7537906-9d0e-4b31-9d76-45936780d55c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/721cf657-76b1-4fdf-868d-d84cb9fa0b14/volumes/kubernetes.io~projected/kube-api-access-bphkr DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7fe8396ac1bb3f74a1e74c4c62f25b12e0eb09e446a3712167e48e0e6154dec4/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/721cf657-76b1-4fdf-868d-d84cb9fa0b14/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/caef3ef2-f9af-4421-866e-61cc4febba1a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/caef3ef2-f9af-4421-866e-61cc4febba1a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ceff794b-7638-49c7-a0f4-495a8cf3d8c7/volumes/kubernetes.io~projected/kube-api-access-qm9bn DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/28f40fee-93d5-47de-89aa-943fa6b4efe7/volumes/kubernetes.io~projected/kube-api-access-q9r9b DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57ee29d6-ad4d-4ed8-ae1a-877732a3737b/volumes/kubernetes.io~projected/kube-api-access-tvqk8 DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5e0b634a45229b1af26839f8414ee1ac9f2c372dd0acb5e8862e567bf6889d4/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad/volumes/kubernetes.io~projected/kube-api-access-p9gkl DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11beabe9c6de3cd56340ccae3df3b8dae40820c3bc49d480260941265241cbdd/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6188415-c8d5-4972-9d6d-f6e3bcf4b079/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd7cda5e-1a4b-45f8-9cab-f96111986c08/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6188415-c8d5-4972-9d6d-f6e3bcf4b079/volumes/kubernetes.io~projected/kube-api-access-8x9z4 DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ac3ab69-eead-4eca-af11-53fec376a6ac/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8ae35879-1912-43b0-a7a5-9ac064ce0dea/volumes/kubernetes.io~projected/kube-api-access-zhzdr DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/11d31a53-ae1b-4d46-aa8c-6c00e89ed539/volumes/kubernetes.io~projected/kube-api-access-77qjb DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68061b2f-50b6-456d-86e7-4b766c7925bc/volumes/kubernetes.io~projected/kube-api-access-qlsmw DeviceMajor:0 DeviceMinor:341 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7537906-9d0e-4b31-9d76-45936780d55c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/20403b1e-e479-45b9-bb85-52cf10725d86/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70a7967543e8a1c5cd506bff801efa08d3a298510b88e1a03def33cee2a138d1/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70748cb186264e6b7d4b61ec14812cd80f483a9b29732084126e01652d7db72e/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee4d07b7-2acc-456e-932b-bfefb2f666df/volumes/kubernetes.io~projected/kube-api-access-ftrxn DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7537906-9d0e-4b31-9d76-45936780d55c/volumes/kubernetes.io~projected/kube-api-access-r5xn9 DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6af08cb7-fa60-491b-88fd-07c1ace45779/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/969393ae-3d0e-4f83-bf03-0396c386ffbf/volumes/kubernetes.io~projected/kube-api-access-8dfz6 DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5557e025-30b7-4315-b7a1-308e166c9a3b/volumes/kubernetes.io~projected/kube-api-access-j4jzx DeviceMajor:0 DeviceMinor:345 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/376eabfad4be616716b578f3918e6800b8fe2eb6a7ce7b84cb32e95810229ef3/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/235840f48c7642fabfb8577e4dc8aae46f92e2b241626f9eda4b30ae41a26152/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/287bd230c933af1b79430cbd50c3f852bdf540f15b5b12a8df5b7853869af3fc/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6af08cb7-fa60-491b-88fd-07c1ace45779/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f641733acf22c32e20702d19d3e0226b46203b223b7985d7ca60947e7541184f/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/633f6b63-a067-470b-bec9-e913e7f28589/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/caef3ef2-f9af-4421-866e-61cc4febba1a/volumes/kubernetes.io~projected/kube-api-access-58fsk DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/452b3edc-e035-446c-9b7b-ee8154f3c6c3/volumes/kubernetes.io~projected/kube-api-access-vlw9l DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3c525ed-5017-482f-ac5c-670002363472/volumes/kubernetes.io~projected/kube-api-access-brtql DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8d10e22-9215-4fc2-8474-5bc68cb57f18/volumes/kubernetes.io~projected/kube-api-access-bfkbg DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9365e876ff21c8086855f6e1563d090b4aebb43c65f350da97ddaf74a361471/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ad4cd78b-b491-4927-80ae-dae026d10e7a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/400470efa985001a73273717eec4096cb361f495ba0b0e9aeaba126ebb524a63/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68061b2f-50b6-456d-86e7-4b766c7925bc/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:334 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/830ddb4e-a8f8-406c-9e89-755bb0d48e0e/volumes/kubernetes.io~projected/kube-api-access-stb92 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ab31aab904cf00ea369247612bac2a6903a09991025c519e691a647ecab581f/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93f0a70e-357e-4d65-b554-42363280bebd/volumes/kubernetes.io~projected/kube-api-access-m4mmt DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91fc483e041346371d853e49b581b082802ddff668b1d2893b35150ba16c0aa2/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/11d31a53-ae1b-4d46-aa8c-6c00e89ed539/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6af08cb7-fa60-491b-88fd-07c1ace45779/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/20403b1e-e479-45b9-bb85-52cf10725d86/volumes/kubernetes.io~projected/kube-api-access-6rxlq DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a7defa5-7e66-4e17-8dc8-e74137e4f054/volumes/kubernetes.io~projected/kube-api-access-gmxmw DeviceMajor:0 DeviceMinor:344 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1107f3e4351949d MacAddress:2a:61:98:f1:24:ce Speed:10000 Mtu:8901} {Name:235840f48c7642f MacAddress:82:06:7c:ff:28:25 Speed:10000 Mtu:8901} {Name:376eabfad4be616 MacAddress:a2:f1:c7:c8:9c:40 Speed:10000 Mtu:8901} {Name:400470efa985001 MacAddress:3e:2d:6d:e6:de:cb Speed:10000 Mtu:8901} {Name:4b378dcf907d51c MacAddress:3a:a0:eb:b8:4e:be Speed:10000 Mtu:8901} {Name:70748cb186264e6 MacAddress:4e:d9:1f:95:ac:09 Speed:10000 Mtu:8901} {Name:70a7967543e8a1c MacAddress:c6:de:a4:fc:2e:0b Speed:10000 Mtu:8901} {Name:74050fbb7ca681a MacAddress:6a:9e:8e:de:18:43 Speed:10000 Mtu:8901} {Name:7a6e72a2cf13bb7 MacAddress:0e:86:4c:29:ff:3a Speed:10000 Mtu:8901} {Name:a5e0b634a45229b MacAddress:ee:be:cf:cc:4d:cd Speed:10000 Mtu:8901} {Name:aaa13a46135b4de MacAddress:f2:cc:82:bc:d0:8d Speed:10000 Mtu:8901} {Name:bb8e5379f323041 MacAddress:de:e7:eb:46:fb:56 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f7:77:57:33 Speed:0 Mtu:9001} {Name:br-int MacAddress:5a:2c:b8:22:c0:e7 Speed:0 Mtu:8901} {Name:df0274035fb0fb4 MacAddress:66:65:d8:e3:a8:d1 Speed:10000 Mtu:8901} {Name:e118e1579fbf034 MacAddress:62:da:3f:93:db:03 Speed:10000 Mtu:8901} {Name:e9365e876ff21c8 MacAddress:02:74:24:81:b4:1d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f7:77:57:33 Speed:0 Mtu:9001} {Name:f641733acf22c32 MacAddress:ca:55:bd:ee:9e:4c Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:3e:a2:e0:ac:2c:31 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:8a:a3:e6:80:90:88 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.986968 5047 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.987029 5047 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.987385 5047 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.987429 5047 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-235.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.987621 5047 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.987632 5047 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.987648 5047 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.987667 5047 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.987794 5047 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.988025 5047 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.988176 5047 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.988224 5047 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.988237 5047 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.988251 5047 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.988264 5047 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.988275 5047 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.989259 5047 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.989273 5047 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.989804 5047 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992072 5047 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992082 5047 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992179 5047 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992359 5047 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992385 5047 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992479 5047 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992491 5047 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992498 5047 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992504 5047 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992509 5047 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992515 5047 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992521 5047 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992526 5047 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992536 5047 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992551 5047 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992563 5047 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992574 5047 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992604 5047 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:04:06.995310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.992612 5047 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:04:06.996609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.996336 5047 server.go:1263] "Started kubelet" Oct 30 19:04:06.996609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.996448 5047 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:04:06.996609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.996526 5047 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:04:06.996742 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.996630 5047 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:04:06.996880 ip-10-0-140-235 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:04:06.997952 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:06.997621 5047 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:04:07.000950 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.000926 5047 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:04:07.002492 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.002356 5047 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:04:07.003036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.002629 5047 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:04:07.003036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.002692 5047 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:04:07.003036 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:07.002737 5047 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-235.ec2.internal\" not found" Oct 30 19:04:07.003036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.002851 5047 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:04:07.003036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.002866 5047 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:04:07.003036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.002879 5047 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:58:28 +0000 UTC" deadline="2028-06-18 18:36:58.33150759 +0000 UTC" Oct 30 19:04:07.003036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.002914 5047 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14327h32m51.328596738s" Oct 30 19:04:07.006538 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.006521 5047 factory.go:55] Registering systemd factory Oct 30 19:04:07.006538 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.006541 5047 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:04:07.008048 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008026 5047 factory.go:153] Registering CRI-O factory Oct 30 19:04:07.008048 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008047 5047 factory.go:223] Registration of the crio container factory successfully Oct 30 19:04:07.008200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008042 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfcde492-9464-42cc-a01c-84d9eec4135a" volumeName="kubernetes.io/projected/bfcde492-9464-42cc-a01c-84d9eec4135a-kube-api-access-9kd76" seLinuxMountContext="" Oct 30 19:04:07.008200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008085 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac3ab69-eead-4eca-af11-53fec376a6ac" volumeName="kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-image-registry-private-configuration" seLinuxMountContext="" Oct 30 19:04:07.008200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008100 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee4d07b7-2acc-456e-932b-bfefb2f666df" volumeName="kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn" seLinuxMountContext="" Oct 30 19:04:07.008200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008108 5047 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:04:07.008200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008113 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a7defa5-7e66-4e17-8dc8-e74137e4f054" volumeName="kubernetes.io/projected/1a7defa5-7e66-4e17-8dc8-e74137e4f054-kube-api-access-gmxmw" seLinuxMountContext="" Oct 30 19:04:07.008200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008127 5047 factory.go:103] Registering Raw factory Oct 30 19:04:07.008200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008127 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac3ab69-eead-4eca-af11-53fec376a6ac" volumeName="kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-certificates" seLinuxMountContext="" Oct 30 19:04:07.008200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008142 5047 manager.go:1199] Started watching for new ooms in manager Oct 30 19:04:07.008200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008141 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6af08cb7-fa60-491b-88fd-07c1ace45779" volumeName="kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-installation-pull-secrets" seLinuxMountContext="" Oct 30 19:04:07.008200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008155 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="830ddb4e-a8f8-406c-9e89-755bb0d48e0e" volumeName="kubernetes.io/projected/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-kube-api-access-stb92" seLinuxMountContext="" Oct 30 19:04:07.008200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008168 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="969393ae-3d0e-4f83-bf03-0396c386ffbf" volumeName="kubernetes.io/projected/969393ae-3d0e-4f83-bf03-0396c386ffbf-kube-api-access-8dfz6" seLinuxMountContext="" Oct 30 19:04:07.008200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008182 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac3ab69-eead-4eca-af11-53fec376a6ac" volumeName="kubernetes.io/empty-dir/3ac3ab69-eead-4eca-af11-53fec376a6ac-ca-trust-extracted" seLinuxMountContext="" Oct 30 19:04:07.008200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008194 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac3ab69-eead-4eca-af11-53fec376a6ac" volumeName="kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-bound-sa-token" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008208 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e626b03-28b0-4840-bbe2-518834311815" volumeName="kubernetes.io/secret/3e626b03-28b0-4840-bbe2-518834311815-cert" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008223 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caef3ef2-f9af-4421-866e-61cc4febba1a" volumeName="kubernetes.io/projected/caef3ef2-f9af-4421-866e-61cc4febba1a-kube-api-access-58fsk" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008236 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ceff794b-7638-49c7-a0f4-495a8cf3d8c7" volumeName="kubernetes.io/configmap/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-daemon-config" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008248 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7537906-9d0e-4b31-9d76-45936780d55c" volumeName="kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-ovnkube-script-lib" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008277 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad" volumeName="kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cni-binary-copy" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008289 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="facfc926-9288-41c4-bdc3-5d8acf446292" volumeName="kubernetes.io/projected/facfc926-9288-41c4-bdc3-5d8acf446292-kube-api-access-m72r5" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008303 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a7defa5-7e66-4e17-8dc8-e74137e4f054" volumeName="kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008318 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="415a1d40-3275-4624-98cd-3df86bbc819a" volumeName="kubernetes.io/projected/415a1d40-3275-4624-98cd-3df86bbc819a-kube-api-access-slnzw" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008331 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="540944ec-348d-46e1-836b-9aeb07d16713" volumeName="kubernetes.io/projected/540944ec-348d-46e1-836b-9aeb07d16713-kube-api-access-6dbws" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008355 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68061b2f-50b6-456d-86e7-4b766c7925bc" volumeName="kubernetes.io/configmap/68061b2f-50b6-456d-86e7-4b766c7925bc-signing-cabundle" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008368 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ceff794b-7638-49c7-a0f4-495a8cf3d8c7" volumeName="kubernetes.io/configmap/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-cni-binary-copy" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008381 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad" volumeName="kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008396 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28f40fee-93d5-47de-89aa-943fa6b4efe7" volumeName="kubernetes.io/configmap/28f40fee-93d5-47de-89aa-943fa6b4efe7-iptables-alerter-script" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008413 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac3ab69-eead-4eca-af11-53fec376a6ac" volumeName="kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-installation-pull-secrets" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008426 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68061b2f-50b6-456d-86e7-4b766c7925bc" volumeName="kubernetes.io/secret/68061b2f-50b6-456d-86e7-4b766c7925bc-signing-key" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008439 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="721cf657-76b1-4fdf-868d-d84cb9fa0b14" volumeName="kubernetes.io/configmap/721cf657-76b1-4fdf-868d-d84cb9fa0b14-config" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008455 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b" volumeName="kubernetes.io/projected/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-kube-api-access-bdw7x" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008469 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcf8833f-ed61-40f4-880c-2645ffae454b" volumeName="kubernetes.io/configmap/bcf8833f-ed61-40f4-880c-2645ffae454b-nginx-conf" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008488 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd7cda5e-1a4b-45f8-9cab-f96111986c08" volumeName="kubernetes.io/configmap/bd7cda5e-1a4b-45f8-9cab-f96111986c08-trusted-ca" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008495 5047 manager.go:319] Starting recovery of all containers Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008499 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac3ab69-eead-4eca-af11-53fec376a6ac" volumeName="kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-trusted-ca" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008526 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="540944ec-348d-46e1-836b-9aeb07d16713" volumeName="kubernetes.io/configmap/540944ec-348d-46e1-836b-9aeb07d16713-telemetry-config" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008538 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="721cf657-76b1-4fdf-868d-d84cb9fa0b14" volumeName="kubernetes.io/secret/721cf657-76b1-4fdf-868d-d84cb9fa0b14-serving-cert" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008557 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7537906-9d0e-4b31-9d76-45936780d55c" volumeName="kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-ovnkube-config" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008572 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7537906-9d0e-4b31-9d76-45936780d55c" volumeName="kubernetes.io/secret/f7537906-9d0e-4b31-9d76-45936780d55c-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008584 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93f0a70e-357e-4d65-b554-42363280bebd" volumeName="kubernetes.io/projected/93f0a70e-357e-4d65-b554-42363280bebd-kube-api-access-m4mmt" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008613 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11d31a53-ae1b-4d46-aa8c-6c00e89ed539" volumeName="kubernetes.io/empty-dir/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-snapshots" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008627 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac3ab69-eead-4eca-af11-53fec376a6ac" volumeName="kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-kube-api-access-zn7mf" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008639 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="415a1d40-3275-4624-98cd-3df86bbc819a" volumeName="kubernetes.io/empty-dir/415a1d40-3275-4624-98cd-3df86bbc819a-tmp-dir" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008651 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5557e025-30b7-4315-b7a1-308e166c9a3b" volumeName="kubernetes.io/empty-dir/5557e025-30b7-4315-b7a1-308e166c9a3b-data-volume" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008663 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="721cf657-76b1-4fdf-868d-d84cb9fa0b14" volumeName="kubernetes.io/projected/721cf657-76b1-4fdf-868d-d84cb9fa0b14-kube-api-access-bphkr" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008678 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b" volumeName="kubernetes.io/configmap/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-config-volume" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008691 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b" volumeName="kubernetes.io/empty-dir/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-tmp-dir" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008703 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11d31a53-ae1b-4d46-aa8c-6c00e89ed539" volumeName="kubernetes.io/configmap/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008715 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a7defa5-7e66-4e17-8dc8-e74137e4f054" volumeName="kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-service-ca-bundle" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008728 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8d10e22-9215-4fc2-8474-5bc68cb57f18" volumeName="kubernetes.io/projected/a8d10e22-9215-4fc2-8474-5bc68cb57f18-kube-api-access-bfkbg" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008741 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b50175d2-e6bb-463a-9602-5bf9e4719a15" volumeName="kubernetes.io/configmap/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-config" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008754 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6188415-c8d5-4972-9d6d-f6e3bcf4b079" volumeName="kubernetes.io/configmap/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-config" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008766 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6188415-c8d5-4972-9d6d-f6e3bcf4b079" volumeName="kubernetes.io/projected/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-kube-api-access-8x9z4" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008779 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3c525ed-5017-482f-ac5c-670002363472" volumeName="kubernetes.io/projected/d3c525ed-5017-482f-ac5c-670002363472-kube-api-access-brtql" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008790 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7537906-9d0e-4b31-9d76-45936780d55c" volumeName="kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-env-overrides" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008802 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28f40fee-93d5-47de-89aa-943fa6b4efe7" volumeName="kubernetes.io/projected/28f40fee-93d5-47de-89aa-943fa6b4efe7-kube-api-access-q9r9b" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008816 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68061b2f-50b6-456d-86e7-4b766c7925bc" volumeName="kubernetes.io/projected/68061b2f-50b6-456d-86e7-4b766c7925bc-kube-api-access-qlsmw" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008830 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad" volumeName="kubernetes.io/projected/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-kube-api-access-p9gkl" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008842 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="830ddb4e-a8f8-406c-9e89-755bb0d48e0e" volumeName="kubernetes.io/configmap/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-serviceca" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008853 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="633f6b63-a067-470b-bec9-e913e7f28589" volumeName="kubernetes.io/configmap/633f6b63-a067-470b-bec9-e913e7f28589-konnectivity-ca" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008866 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd7cda5e-1a4b-45f8-9cab-f96111986c08" volumeName="kubernetes.io/projected/bd7cda5e-1a4b-45f8-9cab-f96111986c08-kube-api-access-w9mcb" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008886 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd7cda5e-1a4b-45f8-9cab-f96111986c08" volumeName="kubernetes.io/secret/bd7cda5e-1a4b-45f8-9cab-f96111986c08-serving-cert" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008899 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caef3ef2-f9af-4421-866e-61cc4febba1a" volumeName="kubernetes.io/empty-dir/caef3ef2-f9af-4421-866e-61cc4febba1a-tmp" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008911 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ceff794b-7638-49c7-a0f4-495a8cf3d8c7" volumeName="kubernetes.io/projected/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-kube-api-access-qm9bn" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008925 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7537906-9d0e-4b31-9d76-45936780d55c" volumeName="kubernetes.io/projected/f7537906-9d0e-4b31-9d76-45936780d55c-kube-api-access-r5xn9" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008936 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11d31a53-ae1b-4d46-aa8c-6c00e89ed539" volumeName="kubernetes.io/configmap/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-service-ca-bundle" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008948 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a7defa5-7e66-4e17-8dc8-e74137e4f054" volumeName="kubernetes.io/empty-dir/1a7defa5-7e66-4e17-8dc8-e74137e4f054-archives-path" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008960 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad4cd78b-b491-4927-80ae-dae026d10e7a" volumeName="kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008971 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20403b1e-e479-45b9-bb85-52cf10725d86" volumeName="kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-stats-auth" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008984 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57ee29d6-ad4d-4ed8-ae1a-877732a3737b" volumeName="kubernetes.io/projected/57ee29d6-ad4d-4ed8-ae1a-877732a3737b-kube-api-access-tvqk8" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.008997 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6af08cb7-fa60-491b-88fd-07c1ace45779" volumeName="kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-certificates" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009024 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6af08cb7-fa60-491b-88fd-07c1ace45779" volumeName="kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-trusted-ca" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009036 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6af08cb7-fa60-491b-88fd-07c1ace45779" volumeName="kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-bound-sa-token" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009048 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ae35879-1912-43b0-a7a5-9ac064ce0dea" volumeName="kubernetes.io/projected/8ae35879-1912-43b0-a7a5-9ac064ce0dea-kube-api-access-zhzdr" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009061 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6188415-c8d5-4972-9d6d-f6e3bcf4b079" volumeName="kubernetes.io/secret/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-serving-cert" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009073 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caef3ef2-f9af-4421-866e-61cc4febba1a" volumeName="kubernetes.io/empty-dir/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-tuned" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009085 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11d31a53-ae1b-4d46-aa8c-6c00e89ed539" volumeName="kubernetes.io/empty-dir/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-tmp" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009100 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11d31a53-ae1b-4d46-aa8c-6c00e89ed539" volumeName="kubernetes.io/projected/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-kube-api-access-77qjb" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009114 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5557e025-30b7-4315-b7a1-308e166c9a3b" volumeName="kubernetes.io/projected/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-api-access-j4jzx" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009125 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6af08cb7-fa60-491b-88fd-07c1ace45779" volumeName="kubernetes.io/empty-dir/6af08cb7-fa60-491b-88fd-07c1ace45779-ca-trust-extracted" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009137 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6af08cb7-fa60-491b-88fd-07c1ace45779" volumeName="kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-kube-api-access-sp2w6" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009150 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6af08cb7-fa60-491b-88fd-07c1ace45779" volumeName="kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-image-registry-private-configuration" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009163 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad" volumeName="kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009206 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd7cda5e-1a4b-45f8-9cab-f96111986c08" volumeName="kubernetes.io/configmap/bd7cda5e-1a4b-45f8-9cab-f96111986c08-config" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009220 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11d31a53-ae1b-4d46-aa8c-6c00e89ed539" volumeName="kubernetes.io/secret/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-serving-cert" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009232 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20403b1e-e479-45b9-bb85-52cf10725d86" volumeName="kubernetes.io/projected/20403b1e-e479-45b9-bb85-52cf10725d86-kube-api-access-6rxlq" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009244 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20403b1e-e479-45b9-bb85-52cf10725d86" volumeName="kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-default-certificate" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009258 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e626b03-28b0-4840-bbe2-518834311815" volumeName="kubernetes.io/projected/3e626b03-28b0-4840-bbe2-518834311815-kube-api-access-wtd7p" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009269 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="452b3edc-e035-446c-9b7b-ee8154f3c6c3" volumeName="kubernetes.io/projected/452b3edc-e035-446c-9b7b-ee8154f3c6c3-kube-api-access-vlw9l" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009283 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="633f6b63-a067-470b-bec9-e913e7f28589" volumeName="kubernetes.io/secret/633f6b63-a067-470b-bec9-e913e7f28589-agent-certs" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009296 5047 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b50175d2-e6bb-463a-9602-5bf9e4719a15" volumeName="kubernetes.io/projected/b50175d2-e6bb-463a-9602-5bf9e4719a15-kube-api-access-2d65z" seLinuxMountContext="" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009307 5047 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:04:07.009566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.009316 5047 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:04:07.030700 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.030670 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/1.log" Oct 30 19:04:07.030853 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.030670 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/1.log" Oct 30 19:04:07.037827 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.037630 5047 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:04:07.040599 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.040575 5047 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:04:07.040599 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.040598 5047 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:04:07.040769 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.040616 5047 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:04:07.040769 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:07.040652 5047 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 19:04:07.044447 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.043802 5047 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:04:07.044447 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.044258 5047 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 19:04:07.044447 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.044430 5047 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:04:07.055475 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.054513 5047 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:04:07.055475 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.055201 5047 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:04:07.057885 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.057864 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/1.log" Oct 30 19:04:07.057961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.057898 5047 generic.go:356] "Generic (PLEG): container finished" podID="bd7cda5e-1a4b-45f8-9cab-f96111986c08" containerID="33635e76a40848e3ec3bc6c36890b08ed835462c6bdf12fcb8d4bee6498cbb95" exitCode=255 Oct 30 19:04:07.086260 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.086228 5047 generic.go:356] "Generic (PLEG): container finished" podID="7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad" containerID="478056f779200b9270c7b20956e870efd2038f99ef4bb794f847106a12fbb09d" exitCode=0 Oct 30 19:04:07.086260 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.086257 5047 generic.go:356] "Generic (PLEG): container finished" podID="7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad" containerID="d13be46182fc6c07d99a744067c1b668fd5bdcb1b4fad8bef35208985cde1066" exitCode=0 Oct 30 19:04:07.086260 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.086266 5047 generic.go:356] "Generic (PLEG): container finished" podID="7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad" containerID="b265bf30eebb720ef475c3fc3343669c94f9f45efa12931a0c76c944e6fd493c" exitCode=0 Oct 30 19:04:07.086466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.086272 5047 generic.go:356] "Generic (PLEG): container finished" podID="7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad" containerID="ea2fbe1913283e26de0d0b3e0f5756cd711eff5aaaf13fed64c8e3fb58f9239e" exitCode=0 Oct 30 19:04:07.086466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.086277 5047 generic.go:356] "Generic (PLEG): container finished" podID="7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad" containerID="3ea4dac667a65d32a0b1107bde80031e155b3580cd78a85983b34148206194b8" exitCode=0 Oct 30 19:04:07.086466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.086282 5047 generic.go:356] "Generic (PLEG): container finished" podID="7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad" containerID="e5c0078982f8695117aa7e623c90d2b8b13d0bec91e123831b9d9493d60c12ca" exitCode=0 Oct 30 19:04:07.093437 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.093103 5047 generic.go:356] "Generic (PLEG): container finished" podID="e3aea26bca5b317ebce58e955350cfdd" containerID="50bacf6f4746f915de68ab49fc167fd35f3fe4d71a4e9d3ad03ecb007d8e3e17" exitCode=0 Oct 30 19:04:07.113624 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.113604 5047 manager.go:324] Recovery completed Oct 30 19:04:07.137499 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.137475 5047 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:04:07.137499 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.137491 5047 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:04:07.137696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.137512 5047 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:04:07.137696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.137639 5047 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 19:04:07.137696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.137651 5047 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 19:04:07.137696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.137673 5047 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 19:04:07.137696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.137679 5047 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 19:04:07.137696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.137684 5047 policy_none.go:50] "Start" Oct 30 19:04:07.137696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.137693 5047 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:04:07.137987 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.137702 5047 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:04:07.137987 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.137843 5047 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 19:04:07.137987 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.137850 5047 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 19:04:07.137987 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.137857 5047 policy_none.go:44] "Start" Oct 30 19:04:07.141047 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:07.141024 5047 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 19:04:07.142400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.142372 5047 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:04:07.145552 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.145538 5047 manager.go:341] "Starting Device Plugin manager" Oct 30 19:04:07.146423 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:07.146401 5047 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:04:07.146499 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.146426 5047 server.go:87] "Starting device plugin registration server" Oct 30 19:04:07.146755 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.146741 5047 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:04:07.146831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.146758 5047 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:04:07.146877 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.146843 5047 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:04:07.146970 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.146941 5047 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:04:07.147046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.146976 5047 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:04:07.147046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.146982 5047 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:04:07.147143 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.147052 5047 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:04:07.146968626Z","UUID":"08635c14-abf1-43e6-ba1c-70e7a2cd758f","Handler":null,"Name":"","Endpoint":""} Oct 30 19:04:07.148566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.148541 5047 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:04:07.148566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.148574 5047 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:04:07.149396 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:07.149375 5047 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:04:07.248312 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.248263 5047 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:04:07.249640 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.249620 5047 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:04:07.249757 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.249651 5047 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:04:07.249757 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.249661 5047 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-235.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:04:07.249757 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.249701 5047 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-235.ec2.internal" Oct 30 19:04:07.279900 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.279868 5047 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-235.ec2.internal" Oct 30 19:04:07.280081 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.279955 5047 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-235.ec2.internal" Oct 30 19:04:07.341628 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.341563 5047 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal"] Oct 30 19:04:07.342319 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.342298 5047 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:04:07.342422 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.342332 5047 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" Oct 30 19:04:07.342422 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.342362 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" event={"ID":"e3aea26bca5b317ebce58e955350cfdd","Type":"ContainerStarted","Data":"890737c9f8c95876cdf4786dbd1fee98d3e378206b04a9495f6c50618318f563"} Oct 30 19:04:07.342509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.342441 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" event={"ID":"e3aea26bca5b317ebce58e955350cfdd","Type":"ContainerDied","Data":"50bacf6f4746f915de68ab49fc167fd35f3fe4d71a4e9d3ad03ecb007d8e3e17"} Oct 30 19:04:07.342509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.342462 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" event={"ID":"e3aea26bca5b317ebce58e955350cfdd","Type":"ContainerStarted","Data":"11beabe9c6de3cd56340ccae3df3b8dae40820c3bc49d480260941265241cbdd"} Oct 30 19:04:07.342509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.342479 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" event={"ID":"b021a22e7072822cf09556c5f1f04826","Type":"ContainerStarted","Data":"6296437556c50144bdb705d00cb452da46161093185cb1906e3c778ad189c014"} Oct 30 19:04:07.342509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.342498 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" event={"ID":"b021a22e7072822cf09556c5f1f04826","Type":"ContainerStarted","Data":"e5d5df8b1241cedec650cf6a47103044b6c2ea856b1f043f15fdd81ec60666f0"} Oct 30 19:04:07.364788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.364715 5047 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:04:07.364788 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:07.364775 5047 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-235.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" Oct 30 19:04:07.364951 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.364944 5047 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:04:07.364986 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:07.364965 5047 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:04:07.446395 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.446342 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3aea26bca5b317ebce58e955350cfdd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal\" (UID: \"e3aea26bca5b317ebce58e955350cfdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:04:07.446551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.446414 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3aea26bca5b317ebce58e955350cfdd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal\" (UID: \"e3aea26bca5b317ebce58e955350cfdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:04:07.446551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.446438 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b021a22e7072822cf09556c5f1f04826-config\") pod \"kube-apiserver-proxy-ip-10-0-140-235.ec2.internal\" (UID: \"b021a22e7072822cf09556c5f1f04826\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" Oct 30 19:04:07.547584 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.547549 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3aea26bca5b317ebce58e955350cfdd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal\" (UID: \"e3aea26bca5b317ebce58e955350cfdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:04:07.547584 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.547585 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b021a22e7072822cf09556c5f1f04826-config\") pod \"kube-apiserver-proxy-ip-10-0-140-235.ec2.internal\" (UID: \"b021a22e7072822cf09556c5f1f04826\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" Oct 30 19:04:07.547771 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.547613 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3aea26bca5b317ebce58e955350cfdd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal\" (UID: \"e3aea26bca5b317ebce58e955350cfdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:04:07.547771 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.547661 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b021a22e7072822cf09556c5f1f04826-config\") pod \"kube-apiserver-proxy-ip-10-0-140-235.ec2.internal\" (UID: \"b021a22e7072822cf09556c5f1f04826\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" Oct 30 19:04:07.547771 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.547661 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3aea26bca5b317ebce58e955350cfdd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal\" (UID: \"e3aea26bca5b317ebce58e955350cfdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:04:07.547771 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.547661 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3aea26bca5b317ebce58e955350cfdd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal\" (UID: \"e3aea26bca5b317ebce58e955350cfdd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:04:07.989074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:07.989038 5047 apiserver.go:52] "Watching apiserver" Oct 30 19:04:08.033496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.033463 5047 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:04:08.034199 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.034168 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wd8lm","openshift-network-console/networking-console-plugin-86c478bb88-nxngq","openshift-image-registry/image-registry-5d4d8bb96d-62jtz","openshift-image-registry/node-ca-tbttr","openshift-ingress-canary/ingress-canary-ccd7w","openshift-network-diagnostics/network-check-source-6f78d59f99-cfdmh","openshift-dns/dns-default-jqrww","openshift-kube-storage-version-migrator/migrator-66df5d496b-lp2xk","openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k","kube-system/global-pull-secret-syncer-hx495","kube-system/kas-connection-checker-685895c7c9-pqg4r","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2","openshift-image-registry/image-registry-5fd9cf966f-mn64l","openshift-service-ca/service-ca-5fffc97bdf-4946j","openshift-console-operator/console-operator-7cd56b8dbc-6h7t5","openshift-cluster-node-tuning-operator/tuned-bdc45","openshift-multus/multus-additional-cni-plugins-k28t4","openshift-insights/insights-runtime-extractor-5rvfx","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l","openshift-ingress/router-default-77d478f789-d787c","openshift-insights/periodic-gathering-hkjk5-l9rq7","openshift-multus/network-metrics-daemon-frj4v","openshift-network-operator/iptables-alerter-wk24l","openshift-dns/node-resolver-xpxgd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal","kube-system/kas-connection-checker-685895c7c9-sfbbt","openshift-kube-storage-version-migrator/migrator-66df5d496b-6zk7f","kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn","kube-system/konnectivity-agent-d26nk","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jdc67","openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s","openshift-ovn-kubernetes/ovnkube-node-7qftp","kube-system/kas-connection-checker-685895c7c9-fs89c","openshift-network-diagnostics/network-check-target-zwhqt","openshift-multus/multus-kgvdt"] Oct 30 19:04:08.036959 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.036938 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:04:08.037664 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.037646 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:04:08.038072 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.038047 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:08.038929 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.038912 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.039331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.039313 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" Oct 30 19:04:08.043395 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.043373 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:04:08.046858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.046833 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.046947 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.046905 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.047784 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.047627 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.048119 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.048064 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.048966 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.048942 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.049139 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.049085 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.049323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.049291 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.049407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.049289 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 19:04:08.049469 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.049431 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.049587 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.049535 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.049877 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.049855 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.049979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.049913 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 19:04:08.049979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.049920 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 19:04:08.049979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.049954 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.050156 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.050001 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8l2hk\"" Oct 30 19:04:08.050156 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.050076 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-c9nhn\"" Oct 30 19:04:08.050156 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.050106 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-t9l5d\"" Oct 30 19:04:08.050156 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.050115 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 19:04:08.050299 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.050210 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rgh44\"" Oct 30 19:04:08.050366 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.050299 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 19:04:08.050430 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.050416 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.050479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.050441 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nft8r\"" Oct 30 19:04:08.050479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.050447 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rzhnr\"" Oct 30 19:04:08.050598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.050582 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-8rqjm\"" Oct 30 19:04:08.050674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.050658 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 19:04:08.050729 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.050675 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 19:04:08.050861 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.050845 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 19:04:08.050918 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.050862 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 19:04:08.050918 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.050913 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.051049 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.050920 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 19:04:08.051184 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.051162 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jqrww" Oct 30 19:04:08.051247 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.051203 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 19:04:08.051407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.051384 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.054133 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.054115 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:08.059788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.059773 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 19:04:08.060430 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.060409 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.060512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.060431 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 19:04:08.060512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.060409 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.060512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.060407 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 19:04:08.060512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.060438 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.060512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.060448 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.060736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.060526 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 19:04:08.060804 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.060781 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.060915 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.060828 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.060915 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.060889 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.061034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.060923 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wbp4l\"" Oct 30 19:04:08.061254 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061241 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.061333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061308 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:04:08.061401 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061335 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.061464 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061419 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.061464 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061428 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 19:04:08.061464 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061436 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:04:08.061464 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061422 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:04:08.061464 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061452 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 19:04:08.061464 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061440 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:04:08.061688 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061476 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.061688 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061471 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.061688 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061486 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 19:04:08.061815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061758 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:04:08.061815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061763 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 19:04:08.061815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061806 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-z6lkw\"" Oct 30 19:04:08.061915 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061872 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4kxxg\"" Oct 30 19:04:08.061915 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061879 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:04:08.061915 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.061882 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:04:08.062035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062019 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 19:04:08.062084 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062060 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 19:04:08.062084 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062068 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 19:04:08.062084 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062072 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-hbw5j\"" Oct 30 19:04:08.062084 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062061 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4bfbh\"" Oct 30 19:04:08.062252 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062113 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 19:04:08.062252 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062188 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:04:08.062252 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062240 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vxbdc\"" Oct 30 19:04:08.062353 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062342 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.062406 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062386 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:04:08.062458 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062414 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.062458 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062431 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.062553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062479 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:04:08.062553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062490 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:04:08.062553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062418 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2sn6r\"" Oct 30 19:04:08.062553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062527 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:04:08.062801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062590 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.062801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062594 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 19:04:08.062801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062624 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2shm8\"" Oct 30 19:04:08.062801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062634 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-s89t7\"" Oct 30 19:04:08.062801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062645 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:04:08.062801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062646 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.062801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062694 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.062801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062699 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:04:08.062801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062698 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9hj6l\"" Oct 30 19:04:08.062801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062707 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fn8hc\"" Oct 30 19:04:08.062801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062715 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.062801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062739 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gp5x6\"" Oct 30 19:04:08.062801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062757 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ghkqg\"" Oct 30 19:04:08.062801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062773 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lgjqt\"" Oct 30 19:04:08.063325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062853 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bwhgk\"" Oct 30 19:04:08.063325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062904 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:04:08.063325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062923 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:04:08.063325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062971 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.063325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.062971 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j7nbz\"" Oct 30 19:04:08.063530 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.063399 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.063530 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.063477 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.063530 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.063521 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:04:08.063530 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.063531 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.065973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.065954 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k9w5x\"" Oct 30 19:04:08.066456 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.066438 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:04:08.066573 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.066549 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bwrqq\"" Oct 30 19:04:08.066686 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.066671 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9ph75\"" Oct 30 19:04:08.066806 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.066793 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.066854 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.066796 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.066991 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.066974 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 19:04:08.067080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.066989 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-h2r74\"" Oct 30 19:04:08.067080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.067071 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 19:04:08.067201 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.067075 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:04:08.067503 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.067484 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 19:04:08.067579 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.067549 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pbgr5\"" Oct 30 19:04:08.067764 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.067751 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:04:08.068066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.068053 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 19:04:08.105377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.105344 5047 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:04:08.107779 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.107754 5047 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:04:08.107902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.107864 5047 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" Oct 30 19:04:08.127590 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.127571 5047 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:04:08.127689 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.127620 5047 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-235.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-235.ec2.internal" Oct 30 19:04:08.127689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.127624 5047 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:04:08.127689 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.127662 5047 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-235.ec2.internal" Oct 30 19:04:08.150222 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150196 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-trusted-ca-bundle\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:08.150337 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150232 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad4cd78b-b491-4927-80ae-dae026d10e7a-dbus\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:04:08.150337 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150257 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:04:08.150337 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150284 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-serviceca\") pod \"node-ca-tbttr\" (UID: \"830ddb4e-a8f8-406c-9e89-755bb0d48e0e\") " pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:04:08.150465 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150328 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-conf-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.150465 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150422 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-run-netns\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.150541 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150491 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-log-socket\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.150541 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150517 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-cni-bin\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.150541 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150535 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-multus-certs\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.150653 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150587 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm9bn\" (UniqueName: \"kubernetes.io/projected/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-kube-api-access-qm9bn\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.150653 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150619 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:08.150653 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150629 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:04:08.150653 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150641 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-trusted-ca-bundle\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:08.150835 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150647 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-serving-cert\") pod \"service-ca-operator-544d749c8c-9ls9s\" (UID: \"c6188415-c8d5-4972-9d6d-f6e3bcf4b079\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" Oct 30 19:04:08.150835 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150694 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-ovnkube-config\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.150835 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150724 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:04:08.150835 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150752 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-service-ca-bundle\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:08.150835 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150804 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9gkl\" (UniqueName: \"kubernetes.io/projected/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-kube-api-access-p9gkl\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:04:08.151106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150864 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6af08cb7-fa60-491b-88fd-07c1ace45779-ca-trust-extracted\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.151106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150893 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-registration-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.151106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150918 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-systemd-units\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.151106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150975 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-service-ca-bundle\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:08.151106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.150991 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6af08cb7-fa60-491b-88fd-07c1ace45779-ca-trust-extracted\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.151106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151030 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77qjb\" (UniqueName: \"kubernetes.io/projected/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-kube-api-access-77qjb\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:04:08.151106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151017 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-ovnkube-config\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.151106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151057 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28f40fee-93d5-47de-89aa-943fa6b4efe7-host-slash\") pod \"iptables-alerter-wk24l\" (UID: \"28f40fee-93d5-47de-89aa-943fa6b4efe7\") " pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:04:08.151106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151063 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-serving-cert\") pod \"service-ca-operator-544d749c8c-9ls9s\" (UID: \"c6188415-c8d5-4972-9d6d-f6e3bcf4b079\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" Oct 30 19:04:08.151106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151084 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfkbg\" (UniqueName: \"kubernetes.io/projected/a8d10e22-9215-4fc2-8474-5bc68cb57f18-kube-api-access-bfkbg\") pod \"kas-connection-checker-685895c7c9-fs89c\" (UID: \"a8d10e22-9215-4fc2-8474-5bc68cb57f18\") " pod="kube-system/kas-connection-checker-685895c7c9-fs89c" Oct 30 19:04:08.151578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151124 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-certificates\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.151578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151147 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-serviceca\") pod \"node-ca-tbttr\" (UID: \"830ddb4e-a8f8-406c-9e89-755bb0d48e0e\") " pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:04:08.151578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151200 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/633f6b63-a067-470b-bec9-e913e7f28589-agent-certs\") pod \"konnectivity-agent-d26nk\" (UID: \"633f6b63-a067-470b-bec9-e913e7f28589\") " pod="kube-system/konnectivity-agent-d26nk" Oct 30 19:04:08.151578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151298 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-hostroot\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.151578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151330 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caef3ef2-f9af-4421-866e-61cc4febba1a-tmp\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.151578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151355 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-host\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.151578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151421 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caef3ef2-f9af-4421-866e-61cc4febba1a-tmp\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.151578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151462 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7537906-9d0e-4b31-9d76-45936780d55c-ovn-node-metrics-cert\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.151578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151538 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cni-binary-copy\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:04:08.151578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151549 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/633f6b63-a067-470b-bec9-e913e7f28589-agent-certs\") pod \"konnectivity-agent-d26nk\" (UID: \"633f6b63-a067-470b-bec9-e913e7f28589\") " pod="kube-system/konnectivity-agent-d26nk" Oct 30 19:04:08.151578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151565 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlw9l\" (UniqueName: \"kubernetes.io/projected/452b3edc-e035-446c-9b7b-ee8154f3c6c3-kube-api-access-vlw9l\") pod \"migrator-66df5d496b-6zk7f\" (UID: \"452b3edc-e035-446c-9b7b-ee8154f3c6c3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6zk7f" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151591 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bphkr\" (UniqueName: \"kubernetes.io/projected/721cf657-76b1-4fdf-868d-d84cb9fa0b14-kube-api-access-bphkr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l\" (UID: \"721cf657-76b1-4fdf-868d-d84cb9fa0b14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151628 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysctl-conf\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151653 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-serving-cert\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151678 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151732 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-etc-selinux\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151740 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cni-binary-copy\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151810 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151845 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/68061b2f-50b6-456d-86e7-4b766c7925bc-signing-key\") pod \"service-ca-5fffc97bdf-4946j\" (UID: \"68061b2f-50b6-456d-86e7-4b766c7925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151876 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7537906-9d0e-4b31-9d76-45936780d55c-ovn-node-metrics-cert\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151888 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-certificates\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151909 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-cnibin\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.151957 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-var-lib-kubelet\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152022 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-slash\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152053 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-cni-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152079 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-kubelet\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152108 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/68061b2f-50b6-456d-86e7-4b766c7925bc-signing-key\") pod \"service-ca-5fffc97bdf-4946j\" (UID: \"68061b2f-50b6-456d-86e7-4b766c7925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152119 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-serving-cert\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152121 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-modprobe-d\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.152145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152149 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-run-ovn-kubernetes\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152192 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152224 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-system-cni-dir\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152251 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-systemd\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152279 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-var-lib-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152311 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-image-registry-private-configuration\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152342 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-installation-pull-secrets\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152417 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rxlq\" (UniqueName: \"kubernetes.io/projected/20403b1e-e479-45b9-bb85-52cf10725d86-kube-api-access-6rxlq\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152434 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-bound-sa-token\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152462 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/68061b2f-50b6-456d-86e7-4b766c7925bc-signing-cabundle\") pod \"service-ca-5fffc97bdf-4946j\" (UID: \"68061b2f-50b6-456d-86e7-4b766c7925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152480 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-tuned\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152487 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-installation-pull-secrets\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152499 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-host\") pod \"node-ca-tbttr\" (UID: \"830ddb4e-a8f8-406c-9e89-755bb0d48e0e\") " pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152501 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-image-registry-private-configuration\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152524 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/721cf657-76b1-4fdf-868d-d84cb9fa0b14-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l\" (UID: \"721cf657-76b1-4fdf-868d-d84cb9fa0b14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152557 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-daemon-config\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152582 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dbws\" (UniqueName: \"kubernetes.io/projected/540944ec-348d-46e1-836b-9aeb07d16713-kube-api-access-6dbws\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152597 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-tuned\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152607 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrxn\" (UniqueName: \"kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn\") pod \"network-check-target-zwhqt\" (UID: \"ee4d07b7-2acc-456e-932b-bfefb2f666df\") " pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152680 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/68061b2f-50b6-456d-86e7-4b766c7925bc-signing-cabundle\") pod \"service-ca-5fffc97bdf-4946j\" (UID: \"68061b2f-50b6-456d-86e7-4b766c7925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152742 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-snapshots\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152772 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d65z\" (UniqueName: \"kubernetes.io/projected/b50175d2-e6bb-463a-9602-5bf9e4719a15-kube-api-access-2d65z\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152790 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152749 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/721cf657-76b1-4fdf-868d-d84cb9fa0b14-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l\" (UID: \"721cf657-76b1-4fdf-868d-d84cb9fa0b14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152805 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-lib-modules\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152818 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-snapshots\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152835 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-config\") pod \"service-ca-operator-544d749c8c-9ls9s\" (UID: \"c6188415-c8d5-4972-9d6d-f6e3bcf4b079\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152819 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-daemon-config\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152866 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhzdr\" (UniqueName: \"kubernetes.io/projected/8ae35879-1912-43b0-a7a5-9ac064ce0dea-kube-api-access-zhzdr\") pod \"kas-connection-checker-685895c7c9-sfbbt\" (UID: \"8ae35879-1912-43b0-a7a5-9ac064ce0dea\") " pod="kube-system/kas-connection-checker-685895c7c9-sfbbt" Oct 30 19:04:08.152902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152907 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-installation-pull-secrets\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152938 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-netns\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.152993 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-etc-kubernetes\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153037 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-config\") pod \"service-ca-operator-544d749c8c-9ls9s\" (UID: \"c6188415-c8d5-4972-9d6d-f6e3bcf4b079\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153059 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-ovn\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153112 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5xn9\" (UniqueName: \"kubernetes.io/projected/f7537906-9d0e-4b31-9d76-45936780d55c-kube-api-access-r5xn9\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153146 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmxmw\" (UniqueName: \"kubernetes.io/projected/1a7defa5-7e66-4e17-8dc8-e74137e4f054-kube-api-access-gmxmw\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153165 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-installation-pull-secrets\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153169 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kd76\" (UniqueName: \"kubernetes.io/projected/bfcde492-9464-42cc-a01c-84d9eec4135a-kube-api-access-9kd76\") pod \"network-check-source-6f78d59f99-cfdmh\" (UID: \"bfcde492-9464-42cc-a01c-84d9eec4135a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cfdmh" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153275 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153423 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-trusted-ca\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153443 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4jzx\" (UniqueName: \"kubernetes.io/projected/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-api-access-j4jzx\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153444 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153461 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-kubernetes\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153497 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdw7x\" (UniqueName: \"kubernetes.io/projected/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-kube-api-access-bdw7x\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153514 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-cni-bin\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153531 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-certificates\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153555 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stb92\" (UniqueName: \"kubernetes.io/projected/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-kube-api-access-stb92\") pod \"node-ca-tbttr\" (UID: \"830ddb4e-a8f8-406c-9e89-755bb0d48e0e\") " pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153590 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-tmp-dir\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153613 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-k8s-cni-cncf-io\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153684 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-tmp-dir\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153711 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x9z4\" (UniqueName: \"kubernetes.io/projected/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-kube-api-access-8x9z4\") pod \"service-ca-operator-544d749c8c-9ls9s\" (UID: \"c6188415-c8d5-4972-9d6d-f6e3bcf4b079\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153732 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-trusted-ca\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.153831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153782 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/415a1d40-3275-4624-98cd-3df86bbc819a-hosts-file\") pod \"node-resolver-xpxgd\" (UID: \"415a1d40-3275-4624-98cd-3df86bbc819a\") " pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153892 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153920 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m72r5\" (UniqueName: \"kubernetes.io/projected/facfc926-9288-41c4-bdc3-5d8acf446292-kube-api-access-m72r5\") pod \"migrator-66df5d496b-lp2xk\" (UID: \"facfc926-9288-41c4-bdc3-5d8acf446292\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lp2xk" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.153978 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/721cf657-76b1-4fdf-868d-d84cb9fa0b14-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l\" (UID: \"721cf657-76b1-4fdf-868d-d84cb9fa0b14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154045 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154089 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysctl-d\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154167 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slnzw\" (UniqueName: \"kubernetes.io/projected/415a1d40-3275-4624-98cd-3df86bbc819a-kube-api-access-slnzw\") pod \"node-resolver-xpxgd\" (UID: \"415a1d40-3275-4624-98cd-3df86bbc819a\") " pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154201 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvqk8\" (UniqueName: \"kubernetes.io/projected/57ee29d6-ad4d-4ed8-ae1a-877732a3737b-kube-api-access-tvqk8\") pod \"kas-connection-checker-685895c7c9-pqg4r\" (UID: \"57ee29d6-ad4d-4ed8-ae1a-877732a3737b\") " pod="kube-system/kas-connection-checker-685895c7c9-pqg4r" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154230 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ac3ab69-eead-4eca-af11-53fec376a6ac-ca-trust-extracted\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154254 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysconfig\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154264 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154281 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-bound-sa-token\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154305 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-sys-fs\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154335 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ac3ab69-eead-4eca-af11-53fec376a6ac-ca-trust-extracted\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154340 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/721cf657-76b1-4fdf-868d-d84cb9fa0b14-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l\" (UID: \"721cf657-76b1-4fdf-868d-d84cb9fa0b14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154357 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154395 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mmt\" (UniqueName: \"kubernetes.io/projected/93f0a70e-357e-4d65-b554-42363280bebd-kube-api-access-m4mmt\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154422 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154447 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-certificates\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154450 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp2w6\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-kube-api-access-sp2w6\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154476 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-cni-netd\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154510 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154578 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-device-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154641 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-tmp\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154669 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cnibin\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154692 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154710 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bcf8833f-ed61-40f4-880c-2645ffae454b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nxngq\" (UID: \"bcf8833f-ed61-40f4-880c-2645ffae454b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" Oct 30 19:04:08.154717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154729 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/415a1d40-3275-4624-98cd-3df86bbc819a-tmp-dir\") pod \"node-resolver-xpxgd\" (UID: \"415a1d40-3275-4624-98cd-3df86bbc819a\") " pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154727 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-tmp\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154751 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154800 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/415a1d40-3275-4624-98cd-3df86bbc819a-tmp-dir\") pod \"node-resolver-xpxgd\" (UID: \"415a1d40-3275-4624-98cd-3df86bbc819a\") " pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154832 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-service-ca-bundle\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154896 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9r9b\" (UniqueName: \"kubernetes.io/projected/28f40fee-93d5-47de-89aa-943fa6b4efe7-kube-api-access-q9r9b\") pod \"iptables-alerter-wk24l\" (UID: \"28f40fee-93d5-47de-89aa-943fa6b4efe7\") " pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154934 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bcf8833f-ed61-40f4-880c-2645ffae454b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nxngq\" (UID: \"bcf8833f-ed61-40f4-880c-2645ffae454b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.154988 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd7cda5e-1a4b-45f8-9cab-f96111986c08-config\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155025 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-service-ca-bundle\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155101 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155118 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155142 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad4cd78b-b491-4927-80ae-dae026d10e7a-kubelet-config\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155272 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd7cda5e-1a4b-45f8-9cab-f96111986c08-config\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155279 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-default-certificate\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155316 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-systemd\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155343 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155384 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nxngq\" (UID: \"bcf8833f-ed61-40f4-880c-2645ffae454b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155423 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn7mf\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-kube-api-access-zn7mf\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155455 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-default-certificate\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155465 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtd7p\" (UniqueName: \"kubernetes.io/projected/3e626b03-28b0-4840-bbe2-518834311815-kube-api-access-wtd7p\") pod \"ingress-canary-ccd7w\" (UID: \"3e626b03-28b0-4840-bbe2-518834311815\") " pod="openshift-ingress-canary/ingress-canary-ccd7w" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155495 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5557e025-30b7-4315-b7a1-308e166c9a3b-crio-socket\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155578 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5557e025-30b7-4315-b7a1-308e166c9a3b-data-volume\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155697 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5557e025-30b7-4315-b7a1-308e166c9a3b-data-volume\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:08.155723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155714 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fsk\" (UniqueName: \"kubernetes.io/projected/caef3ef2-f9af-4421-866e-61cc4febba1a-kube-api-access-58fsk\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155738 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155754 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9mcb\" (UniqueName: \"kubernetes.io/projected/bd7cda5e-1a4b-45f8-9cab-f96111986c08-kube-api-access-w9mcb\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155779 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/633f6b63-a067-470b-bec9-e913e7f28589-konnectivity-ca\") pod \"konnectivity-agent-d26nk\" (UID: \"633f6b63-a067-470b-bec9-e913e7f28589\") " pod="kube-system/konnectivity-agent-d26nk" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155816 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-socket-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155845 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlsmw\" (UniqueName: \"kubernetes.io/projected/68061b2f-50b6-456d-86e7-4b766c7925bc-kube-api-access-qlsmw\") pod \"service-ca-5fffc97bdf-4946j\" (UID: \"68061b2f-50b6-456d-86e7-4b766c7925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155874 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dfz6\" (UniqueName: \"kubernetes.io/projected/969393ae-3d0e-4f83-bf03-0396c386ffbf-kube-api-access-8dfz6\") pod \"volume-data-source-validator-58c9f5f46d-jdc67\" (UID: \"969393ae-3d0e-4f83-bf03-0396c386ffbf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jdc67" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155907 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-sys\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155928 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd7cda5e-1a4b-45f8-9cab-f96111986c08-serving-cert\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155950 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-cni-binary-copy\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155965 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/633f6b63-a067-470b-bec9-e913e7f28589-konnectivity-ca\") pod \"konnectivity-agent-d26nk\" (UID: \"633f6b63-a067-470b-bec9-e913e7f28589\") " pod="kube-system/konnectivity-agent-d26nk" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.155985 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-etc-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156045 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-node-log\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156049 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad4cd78b-b491-4927-80ae-dae026d10e7a-original-pull-secret\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156065 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-trusted-ca\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156089 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-stats-auth\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156124 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/540944ec-348d-46e1-836b-9aeb07d16713-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156190 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-cni-binary-copy\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156206 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-env-overrides\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156243 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd7cda5e-1a4b-45f8-9cab-f96111986c08-serving-cert\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156251 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28f40fee-93d5-47de-89aa-943fa6b4efe7-iptables-alerter-script\") pod \"iptables-alerter-wk24l\" (UID: \"28f40fee-93d5-47de-89aa-943fa6b4efe7\") " pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156287 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e626b03-28b0-4840-bbe2-518834311815-cert\") pod \"ingress-canary-ccd7w\" (UID: \"3e626b03-28b0-4840-bbe2-518834311815\") " pod="openshift-ingress-canary/ingress-canary-ccd7w" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156310 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-stats-auth\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156313 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-system-cni-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156345 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-ovnkube-script-lib\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156372 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-env-overrides\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156411 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-config-volume\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156445 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/540944ec-348d-46e1-836b-9aeb07d16713-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156435 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd7cda5e-1a4b-45f8-9cab-f96111986c08-trusted-ca\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156486 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-trusted-ca\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156505 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28f40fee-93d5-47de-89aa-943fa6b4efe7-iptables-alerter-script\") pod \"iptables-alerter-wk24l\" (UID: \"28f40fee-93d5-47de-89aa-943fa6b4efe7\") " pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156511 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e626b03-28b0-4840-bbe2-518834311815-cert\") pod \"ingress-canary-ccd7w\" (UID: \"3e626b03-28b0-4840-bbe2-518834311815\") " pod="openshift-ingress-canary/ingress-canary-ccd7w" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156556 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1a7defa5-7e66-4e17-8dc8-e74137e4f054-archives-path\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156588 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-image-registry-private-configuration\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156612 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1a7defa5-7e66-4e17-8dc8-e74137e4f054-archives-path\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156621 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-socket-dir-parent\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156649 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-run\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156665 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-config-volume\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156673 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7537906-9d0e-4b31-9d76-45936780d55c-ovnkube-script-lib\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156675 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-kubelet\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156719 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brtql\" (UniqueName: \"kubernetes.io/projected/d3c525ed-5017-482f-ac5c-670002363472-kube-api-access-brtql\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.156747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156744 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-cni-multus\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.157886 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156776 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-image-registry-private-configuration\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.157886 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156798 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.157886 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.156844 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd7cda5e-1a4b-45f8-9cab-f96111986c08-trusted-ca\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:04:08.168305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.168281 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9gkl\" (UniqueName: \"kubernetes.io/projected/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-kube-api-access-p9gkl\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:04:08.168305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.168297 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlw9l\" (UniqueName: \"kubernetes.io/projected/452b3edc-e035-446c-9b7b-ee8154f3c6c3-kube-api-access-vlw9l\") pod \"migrator-66df5d496b-6zk7f\" (UID: \"452b3edc-e035-446c-9b7b-ee8154f3c6c3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6zk7f" Oct 30 19:04:08.168455 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.168288 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm9bn\" (UniqueName: \"kubernetes.io/projected/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-kube-api-access-qm9bn\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.168455 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.168281 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bphkr\" (UniqueName: \"kubernetes.io/projected/721cf657-76b1-4fdf-868d-d84cb9fa0b14-kube-api-access-bphkr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l\" (UID: \"721cf657-76b1-4fdf-868d-d84cb9fa0b14\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" Oct 30 19:04:08.168455 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.168430 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfkbg\" (UniqueName: \"kubernetes.io/projected/a8d10e22-9215-4fc2-8474-5bc68cb57f18-kube-api-access-bfkbg\") pod \"kas-connection-checker-685895c7c9-fs89c\" (UID: \"a8d10e22-9215-4fc2-8474-5bc68cb57f18\") " pod="kube-system/kas-connection-checker-685895c7c9-fs89c" Oct 30 19:04:08.168554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.168503 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77qjb\" (UniqueName: \"kubernetes.io/projected/11d31a53-ae1b-4d46-aa8c-6c00e89ed539-kube-api-access-77qjb\") pod \"insights-operator-7c4986d449-wd8lm\" (UID: \"11d31a53-ae1b-4d46-aa8c-6c00e89ed539\") " pod="openshift-insights/insights-operator-7c4986d449-wd8lm" Oct 30 19:04:08.169144 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.169124 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftrxn\" (UniqueName: \"kubernetes.io/projected/ee4d07b7-2acc-456e-932b-bfefb2f666df-kube-api-access-ftrxn\") pod \"network-check-target-zwhqt\" (UID: \"ee4d07b7-2acc-456e-932b-bfefb2f666df\") " pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:04:08.170346 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.170326 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stb92\" (UniqueName: \"kubernetes.io/projected/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-kube-api-access-stb92\") pod \"node-ca-tbttr\" (UID: \"830ddb4e-a8f8-406c-9e89-755bb0d48e0e\") " pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:04:08.170425 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.170378 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x9z4\" (UniqueName: \"kubernetes.io/projected/c6188415-c8d5-4972-9d6d-f6e3bcf4b079-kube-api-access-8x9z4\") pod \"service-ca-operator-544d749c8c-9ls9s\" (UID: \"c6188415-c8d5-4972-9d6d-f6e3bcf4b079\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" Oct 30 19:04:08.170713 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.170692 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdw7x\" (UniqueName: \"kubernetes.io/projected/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-kube-api-access-bdw7x\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:04:08.170940 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.170922 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4jzx\" (UniqueName: \"kubernetes.io/projected/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-api-access-j4jzx\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:08.170998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.170974 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d65z\" (UniqueName: \"kubernetes.io/projected/b50175d2-e6bb-463a-9602-5bf9e4719a15-kube-api-access-2d65z\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:04:08.171119 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.171102 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhzdr\" (UniqueName: \"kubernetes.io/projected/8ae35879-1912-43b0-a7a5-9ac064ce0dea-kube-api-access-zhzdr\") pod \"kas-connection-checker-685895c7c9-sfbbt\" (UID: \"8ae35879-1912-43b0-a7a5-9ac064ce0dea\") " pod="kube-system/kas-connection-checker-685895c7c9-sfbbt" Oct 30 19:04:08.171641 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.171623 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dbws\" (UniqueName: \"kubernetes.io/projected/540944ec-348d-46e1-836b-9aeb07d16713-kube-api-access-6dbws\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:04:08.172092 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.172073 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m72r5\" (UniqueName: \"kubernetes.io/projected/facfc926-9288-41c4-bdc3-5d8acf446292-kube-api-access-m72r5\") pod \"migrator-66df5d496b-lp2xk\" (UID: \"facfc926-9288-41c4-bdc3-5d8acf446292\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lp2xk" Oct 30 19:04:08.172537 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.172518 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kd76\" (UniqueName: \"kubernetes.io/projected/bfcde492-9464-42cc-a01c-84d9eec4135a-kube-api-access-9kd76\") pod \"network-check-source-6f78d59f99-cfdmh\" (UID: \"bfcde492-9464-42cc-a01c-84d9eec4135a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cfdmh" Oct 30 19:04:08.172678 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.172659 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5xn9\" (UniqueName: \"kubernetes.io/projected/f7537906-9d0e-4b31-9d76-45936780d55c-kube-api-access-r5xn9\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.173234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.173203 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-bound-sa-token\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.173234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.173230 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvqk8\" (UniqueName: \"kubernetes.io/projected/57ee29d6-ad4d-4ed8-ae1a-877732a3737b-kube-api-access-tvqk8\") pod \"kas-connection-checker-685895c7c9-pqg4r\" (UID: \"57ee29d6-ad4d-4ed8-ae1a-877732a3737b\") " pod="kube-system/kas-connection-checker-685895c7c9-pqg4r" Oct 30 19:04:08.173370 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.173239 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rxlq\" (UniqueName: \"kubernetes.io/projected/20403b1e-e479-45b9-bb85-52cf10725d86-kube-api-access-6rxlq\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:08.173370 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.173283 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp2w6\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-kube-api-access-sp2w6\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.173370 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.173285 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-bound-sa-token\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.173370 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.173289 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slnzw\" (UniqueName: \"kubernetes.io/projected/415a1d40-3275-4624-98cd-3df86bbc819a-kube-api-access-slnzw\") pod \"node-resolver-xpxgd\" (UID: \"415a1d40-3275-4624-98cd-3df86bbc819a\") " pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:04:08.175736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.175709 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtd7p\" (UniqueName: \"kubernetes.io/projected/3e626b03-28b0-4840-bbe2-518834311815-kube-api-access-wtd7p\") pod \"ingress-canary-ccd7w\" (UID: \"3e626b03-28b0-4840-bbe2-518834311815\") " pod="openshift-ingress-canary/ingress-canary-ccd7w" Oct 30 19:04:08.175873 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.175799 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58fsk\" (UniqueName: \"kubernetes.io/projected/caef3ef2-f9af-4421-866e-61cc4febba1a-kube-api-access-58fsk\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.176109 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.176092 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9mcb\" (UniqueName: \"kubernetes.io/projected/bd7cda5e-1a4b-45f8-9cab-f96111986c08-kube-api-access-w9mcb\") pod \"console-operator-7cd56b8dbc-6h7t5\" (UID: \"bd7cda5e-1a4b-45f8-9cab-f96111986c08\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:04:08.176178 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.176113 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn7mf\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-kube-api-access-zn7mf\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.176178 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.176115 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brtql\" (UniqueName: \"kubernetes.io/projected/d3c525ed-5017-482f-ac5c-670002363472-kube-api-access-brtql\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.176244 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.176229 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9r9b\" (UniqueName: \"kubernetes.io/projected/28f40fee-93d5-47de-89aa-943fa6b4efe7-kube-api-access-q9r9b\") pod \"iptables-alerter-wk24l\" (UID: \"28f40fee-93d5-47de-89aa-943fa6b4efe7\") " pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:04:08.176317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.176300 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlsmw\" (UniqueName: \"kubernetes.io/projected/68061b2f-50b6-456d-86e7-4b766c7925bc-kube-api-access-qlsmw\") pod \"service-ca-5fffc97bdf-4946j\" (UID: \"68061b2f-50b6-456d-86e7-4b766c7925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4946j" Oct 30 19:04:08.176574 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.176556 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dfz6\" (UniqueName: \"kubernetes.io/projected/969393ae-3d0e-4f83-bf03-0396c386ffbf-kube-api-access-8dfz6\") pod \"volume-data-source-validator-58c9f5f46d-jdc67\" (UID: \"969393ae-3d0e-4f83-bf03-0396c386ffbf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jdc67" Oct 30 19:04:08.176609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.176563 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmxmw\" (UniqueName: \"kubernetes.io/projected/1a7defa5-7e66-4e17-8dc8-e74137e4f054-kube-api-access-gmxmw\") pod \"periodic-gathering-hkjk5-l9rq7\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:08.177488 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.177464 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4mmt\" (UniqueName: \"kubernetes.io/projected/93f0a70e-357e-4d65-b554-42363280bebd-kube-api-access-m4mmt\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:04:08.257409 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257328 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-device-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.257409 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257372 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cnibin\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:04:08.257595 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257417 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:04:08.257595 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257435 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-cnibin\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:04:08.257595 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257473 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-device-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.257595 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257484 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:08.257595 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257509 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:04:08.257595 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257535 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad4cd78b-b491-4927-80ae-dae026d10e7a-kubelet-config\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:04:08.257595 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257560 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-systemd\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.257595 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.257572 5047 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:04:08.257595 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257587 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257598 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257613 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nxngq\" (UID: \"bcf8833f-ed61-40f4-880c-2645ffae454b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257647 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad4cd78b-b491-4927-80ae-dae026d10e7a-kubelet-config\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.257675 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs podName:93f0a70e-357e-4d65-b554-42363280bebd nodeName:}" failed. No retries permitted until 2026-10-30 19:04:08.75762937 +0000 UTC m=+1.832084723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs") pod "network-metrics-daemon-frj4v" (UID: "93f0a70e-357e-4d65-b554-42363280bebd") : secret "metrics-daemon-secret" not found Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257672 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-systemd\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257694 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5557e025-30b7-4315-b7a1-308e166c9a3b-crio-socket\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257733 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-socket-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257762 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-sys\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257778 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5557e025-30b7-4315-b7a1-308e166c9a3b-crio-socket\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257790 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-etc-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257830 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-etc-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.257842 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle podName:20403b1e-e479-45b9-bb85-52cf10725d86 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:08.757785209 +0000 UTC m=+1.832240578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle") pod "router-default-77d478f789-d787c" (UID: "20403b1e-e479-45b9-bb85-52cf10725d86") : configmap references non-existent config key: service-ca.crt Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257873 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-socket-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257885 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-sys\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257894 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-node-log\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257926 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-system-cni-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257962 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-socket-dir-parent\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257981 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-run\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257964 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-node-log\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.257995 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-kubelet\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258038 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-socket-dir-parent\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258039 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-system-cni-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.258044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258064 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-cni-multus\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258068 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-run\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258079 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-kubelet\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258089 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258040 5047 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258122 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-cni-multus\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258135 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad4cd78b-b491-4927-80ae-dae026d10e7a-dbus\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258158 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258162 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-conf-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258180 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad4cd78b-b491-4927-80ae-dae026d10e7a-dbus\") pod \"global-pull-secret-syncer-hx495\" (UID: \"ad4cd78b-b491-4927-80ae-dae026d10e7a\") " pod="kube-system/global-pull-secret-syncer-hx495" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258196 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-run-netns\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258222 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-log-socket\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258242 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-cni-bin\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258199 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-conf-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258257 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-multus-certs\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258280 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-multus-certs\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258295 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-log-socket\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258318 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258346 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-cni-bin\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258355 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258327 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-run-netns\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258398 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-registration-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258423 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-systemd-units\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258448 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28f40fee-93d5-47de-89aa-943fa6b4efe7-host-slash\") pod \"iptables-alerter-wk24l\" (UID: \"28f40fee-93d5-47de-89aa-943fa6b4efe7\") " pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258484 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-systemd-units\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.258458 5047 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258498 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-registration-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258527 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-hostroot\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258494 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28f40fee-93d5-47de-89aa-943fa6b4efe7-host-slash\") pod \"iptables-alerter-wk24l\" (UID: \"28f40fee-93d5-47de-89aa-943fa6b4efe7\") " pod="openshift-network-operator/iptables-alerter-wk24l" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258490 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-hostroot\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.258566 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls podName:a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:08.75852755 +0000 UTC m=+1.832982904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls") pod "dns-default-jqrww" (UID: "a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b") : secret "dns-default-metrics-tls" not found Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.258610 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-host\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.259030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259027 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysctl-conf\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259084 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259117 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-etc-selinux\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259146 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259181 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-cnibin\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259212 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-var-lib-kubelet\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259218 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5557e025-30b7-4315-b7a1-308e166c9a3b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259242 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-slash\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259321 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-cnibin\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259331 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-var-lib-kubelet\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259362 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-cni-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259392 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysctl-conf\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.259413 5047 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259457 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-slash\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259397 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-kubelet\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259522 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-etc-selinux\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.259541 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs podName:20403b1e-e479-45b9-bb85-52cf10725d86 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:08.759502433 +0000 UTC m=+1.833957789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs") pod "router-default-77d478f789-d787c" (UID: "20403b1e-e479-45b9-bb85-52cf10725d86") : secret "router-metrics-certs-default" not found Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259562 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-modprobe-d\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259561 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-kubelet\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.259614 5047 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259638 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-host\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.259671 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls podName:b50175d2-e6bb-463a-9602-5bf9e4719a15 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:08.759658441 +0000 UTC m=+1.834113788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lx6v2" (UID: "b50175d2-e6bb-463a-9602-5bf9e4719a15") : secret "samples-operator-tls" not found Oct 30 19:04:08.260374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.259715 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-multus-cni-dir\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.261316 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.260844 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/540944ec-348d-46e1-836b-9aeb07d16713-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x4r6k\" (UID: \"540944ec-348d-46e1-836b-9aeb07d16713\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:04:08.264136 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264110 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-run-ovn-kubernetes\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.264280 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264239 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-modprobe-d\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.264329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264271 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.264329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264260 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-run-ovn-kubernetes\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.264388 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264349 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-system-cni-dir\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:04:08.264388 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264289 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.264388 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264378 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-systemd\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.264496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264411 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-var-lib-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.264496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264415 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad-system-cni-dir\") pod \"multus-additional-cni-plugins-k28t4\" (UID: \"7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad\") " pod="openshift-multus/multus-additional-cni-plugins-k28t4" Oct 30 19:04:08.264496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264444 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-systemd\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.264496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264445 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-host\") pod \"node-ca-tbttr\" (UID: \"830ddb4e-a8f8-406c-9e89-755bb0d48e0e\") " pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:04:08.264496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264470 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/830ddb4e-a8f8-406c-9e89-755bb0d48e0e-host\") pod \"node-ca-tbttr\" (UID: \"830ddb4e-a8f8-406c-9e89-755bb0d48e0e\") " pod="openshift-image-registry/node-ca-tbttr" Oct 30 19:04:08.264665 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264505 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-var-lib-openvswitch\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.264901 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264515 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.265026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264921 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-lib-modules\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.265026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264950 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-netns\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.265026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264973 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-etc-kubernetes\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.265026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.264994 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-ovn\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.264926 5047 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265035 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-kubernetes\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265038 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bcf8833f-ed61-40f4-880c-2645ffae454b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nxngq\" (UID: \"bcf8833f-ed61-40f4-880c-2645ffae454b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265059 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-cni-bin\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265060 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-lib-modules\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265081 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-run-ovn\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265044 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-netns\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265082 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-etc-kubernetes\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265108 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-kubernetes\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265099 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-k8s-cni-cncf-io\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.265046 5047 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4d8bb96d-62jtz: secret "image-registry-tls" not found Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265136 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/415a1d40-3275-4624-98cd-3df86bbc819a-hosts-file\") pod \"node-resolver-xpxgd\" (UID: \"415a1d40-3275-4624-98cd-3df86bbc819a\") " pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265137 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-run-k8s-cni-cncf-io\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265156 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265118 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceff794b-7638-49c7-a0f4-495a8cf3d8c7-host-var-lib-cni-bin\") pod \"multus-kgvdt\" (UID: \"ceff794b-7638-49c7-a0f4-495a8cf3d8c7\") " pod="openshift-multus/multus-kgvdt" Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.265181 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls podName:6af08cb7-fa60-491b-88fd-07c1ace45779 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:08.765167486 +0000 UTC m=+1.839622837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls") pod "image-registry-5d4d8bb96d-62jtz" (UID: "6af08cb7-fa60-491b-88fd-07c1ace45779") : secret "image-registry-tls" not found Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265190 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/415a1d40-3275-4624-98cd-3df86bbc819a-hosts-file\") pod \"node-resolver-xpxgd\" (UID: \"415a1d40-3275-4624-98cd-3df86bbc819a\") " pod="openshift-dns/node-resolver-xpxgd" Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265204 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysctl-d\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.265213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265210 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.265813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265229 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysconfig\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.265813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265251 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-sys-fs\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.265813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265286 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:08.265813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265294 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysconfig\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.265813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265299 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3c525ed-5017-482f-ac5c-670002363472-sys-fs\") pod \"aws-ebs-csi-driver-node-cm7cn\" (UID: \"d3c525ed-5017-482f-ac5c-670002363472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cm7cn" Oct 30 19:04:08.265813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265313 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.265813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265350 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-cni-netd\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.265813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265364 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caef3ef2-f9af-4421-866e-61cc4febba1a-etc-sysctl-d\") pod \"tuned-bdc45\" (UID: \"caef3ef2-f9af-4421-866e-61cc4febba1a\") " pod="openshift-cluster-node-tuning-operator/tuned-bdc45" Oct 30 19:04:08.265813 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.265384 5047 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:04:08.265813 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.265392 5047 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:04:08.265813 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.265402 5047 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd9cf966f-mn64l: secret "image-registry-tls" not found Oct 30 19:04:08.265813 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.265430 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls podName:3ac3ab69-eead-4eca-af11-53fec376a6ac nodeName:}" failed. No retries permitted until 2026-10-30 19:04:08.765420653 +0000 UTC m=+1.839876012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls") pod "image-registry-5fd9cf966f-mn64l" (UID: "3ac3ab69-eead-4eca-af11-53fec376a6ac") : secret "image-registry-tls" not found Oct 30 19:04:08.265813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.265427 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7537906-9d0e-4b31-9d76-45936780d55c-host-cni-netd\") pod \"ovnkube-node-7qftp\" (UID: \"f7537906-9d0e-4b31-9d76-45936780d55c\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.265813 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.265492 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls podName:5557e025-30b7-4315-b7a1-308e166c9a3b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:08.765471012 +0000 UTC m=+1.839926355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rvfx" (UID: "5557e025-30b7-4315-b7a1-308e166c9a3b") : secret "insights-runtime-extractor-tls" not found Oct 30 19:04:08.336884 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.336854 5047 scope.go:122] "RemoveContainer" containerID="33635e76a40848e3ec3bc6c36890b08ed835462c6bdf12fcb8d4bee6498cbb95" Oct 30 19:04:08.337229 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.337210 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" Oct 30 19:04:08.339681 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.339660 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" Oct 30 19:04:08.470400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.470375 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xpxgd_415a1d40-3275-4624-98cd-3df86bbc819a/dns-node-resolver/0.log" Oct 30 19:04:08.664936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.664910 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k"] Oct 30 19:04:08.667503 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:08.667463 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540944ec_348d_46e1_836b_9aeb07d16713.slice/crio-7cdbccc592ad68369597b0b86794355c4642a7dd754df46622830ccea6217862 WatchSource:0}: Error finding container 7cdbccc592ad68369597b0b86794355c4642a7dd754df46622830ccea6217862: Status 404 returned error can't find the container with id 7cdbccc592ad68369597b0b86794355c4642a7dd754df46622830ccea6217862 Oct 30 19:04:08.669939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.669922 5047 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:04:08.683910 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.683889 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:04:08.686115 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.686092 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-nxngq"] Oct 30 19:04:08.688672 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:08.688651 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf8833f_ed61_40f4_880c_2645ffae454b.slice/crio-297e45714a3b4b9ed23c554d3fe7e4cd9f5ec035708a5fd282f1de9f0cbd249a WatchSource:0}: Error finding container 297e45714a3b4b9ed23c554d3fe7e4cd9f5ec035708a5fd282f1de9f0cbd249a: Status 404 returned error can't find the container with id 297e45714a3b4b9ed23c554d3fe7e4cd9f5ec035708a5fd282f1de9f0cbd249a Oct 30 19:04:08.701440 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.701412 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:08.769860 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.769774 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:08.769860 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.769816 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:04:08.770116 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.769879 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:08.770116 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.769925 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:08.770116 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.769948 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:08.770116 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.769978 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:04:08.770116 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.770028 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:08.770116 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.770032 5047 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:04:08.770116 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.770083 5047 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:04:08.770116 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.770089 5047 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:04:08.770116 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.770104 5047 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd9cf966f-mn64l: secret "image-registry-tls" not found Oct 30 19:04:08.770453 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.770039 5047 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:04:08.770453 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.770140 5047 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4d8bb96d-62jtz: secret "image-registry-tls" not found Oct 30 19:04:08.770453 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.770102 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls podName:b50175d2-e6bb-463a-9602-5bf9e4719a15 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:09.770084237 +0000 UTC m=+2.844539597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lx6v2" (UID: "b50175d2-e6bb-463a-9602-5bf9e4719a15") : secret "samples-operator-tls" not found Oct 30 19:04:08.770453 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.770032 5047 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:04:08.770453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.770167 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:04:08.770453 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.770174 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle podName:20403b1e-e479-45b9-bb85-52cf10725d86 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:09.770157211 +0000 UTC m=+2.844612552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle") pod "router-default-77d478f789-d787c" (UID: "20403b1e-e479-45b9-bb85-52cf10725d86") : configmap references non-existent config key: service-ca.crt Oct 30 19:04:08.770453 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.770198 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls podName:5557e025-30b7-4315-b7a1-308e166c9a3b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:09.770187809 +0000 UTC m=+2.844643147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rvfx" (UID: "5557e025-30b7-4315-b7a1-308e166c9a3b") : secret "insights-runtime-extractor-tls" not found Oct 30 19:04:08.770453 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.770218 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls podName:6af08cb7-fa60-491b-88fd-07c1ace45779 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:09.770207313 +0000 UTC m=+2.844662656 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls") pod "image-registry-5d4d8bb96d-62jtz" (UID: "6af08cb7-fa60-491b-88fd-07c1ace45779") : secret "image-registry-tls" not found Oct 30 19:04:08.770453 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.770234 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs podName:93f0a70e-357e-4d65-b554-42363280bebd nodeName:}" failed. No retries permitted until 2026-10-30 19:04:09.770225994 +0000 UTC m=+2.844681330 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs") pod "network-metrics-daemon-frj4v" (UID: "93f0a70e-357e-4d65-b554-42363280bebd") : secret "metrics-daemon-secret" not found Oct 30 19:04:08.770453 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:08.770250 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls podName:3ac3ab69-eead-4eca-af11-53fec376a6ac nodeName:}" failed. No retries permitted until 2026-10-30 19:04:09.770243048 +0000 UTC m=+2.844698384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls") pod "image-registry-5fd9cf966f-mn64l" (UID: "3ac3ab69-eead-4eca-af11-53fec376a6ac") : secret "image-registry-tls" not found Oct 30 19:04:08.772190 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.772175 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20403b1e-e479-45b9-bb85-52cf10725d86-metrics-certs\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:08.772230 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.772219 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b-metrics-tls\") pod \"dns-default-jqrww\" (UID: \"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b\") " pod="openshift-dns/dns-default-jqrww" Oct 30 19:04:08.952353 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:08.952316 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jqrww" Oct 30 19:04:09.111635 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:09.111592 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" event={"ID":"bcf8833f-ed61-40f4-880c-2645ffae454b","Type":"ContainerStarted","Data":"297e45714a3b4b9ed23c554d3fe7e4cd9f5ec035708a5fd282f1de9f0cbd249a"} Oct 30 19:04:09.112701 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:09.112677 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" event={"ID":"540944ec-348d-46e1-836b-9aeb07d16713","Type":"ContainerStarted","Data":"7cdbccc592ad68369597b0b86794355c4642a7dd754df46622830ccea6217862"} Oct 30 19:04:09.114206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:09.114188 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:04:09.114591 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:09.114569 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/1.log" Oct 30 19:04:09.114670 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:09.114601 5047 generic.go:356] "Generic (PLEG): container finished" podID="bd7cda5e-1a4b-45f8-9cab-f96111986c08" containerID="673c43d635dd91bc96e71ed74ed420916c8405959101bc1bfdb3202aac48722b" exitCode=255 Oct 30 19:04:09.114717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:09.114686 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" event={"ID":"bd7cda5e-1a4b-45f8-9cab-f96111986c08","Type":"ContainerDied","Data":"673c43d635dd91bc96e71ed74ed420916c8405959101bc1bfdb3202aac48722b"} Oct 30 19:04:09.114779 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:09.114740 5047 scope.go:122] "RemoveContainer" containerID="33635e76a40848e3ec3bc6c36890b08ed835462c6bdf12fcb8d4bee6498cbb95" Oct 30 19:04:09.115453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:09.115434 5047 scope.go:122] "RemoveContainer" containerID="673c43d635dd91bc96e71ed74ed420916c8405959101bc1bfdb3202aac48722b" Oct 30 19:04:09.115682 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:09.115660 5047 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6h7t5_openshift-console-operator(bd7cda5e-1a4b-45f8-9cab-f96111986c08)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" podUID="bd7cda5e-1a4b-45f8-9cab-f96111986c08" Oct 30 19:04:09.267838 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:09.267809 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jqrww"] Oct 30 19:04:09.270928 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:09.270906 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tbttr_830ddb4e-a8f8-406c-9e89-755bb0d48e0e/node-ca/0.log" Oct 30 19:04:09.272326 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:09.272301 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b2582d_0fc5_4ab8_94aa_6ac1310b3f6b.slice/crio-7d305de0af43dbecaf4384c4ceb75a4fd3762f16efb8215270e2c2c1ae252339 WatchSource:0}: Error finding container 7d305de0af43dbecaf4384c4ceb75a4fd3762f16efb8215270e2c2c1ae252339: Status 404 returned error can't find the container with id 7d305de0af43dbecaf4384c4ceb75a4fd3762f16efb8215270e2c2c1ae252339 Oct 30 19:04:09.778423 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:09.778374 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:09.778423 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:09.778425 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:09.778652 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:09.778461 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:04:09.778652 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:09.778492 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:09.778652 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:09.778555 5047 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:04:09.778652 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:09.778569 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:04:09.778652 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:09.778596 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:09.778652 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:09.778634 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls podName:5557e025-30b7-4315-b7a1-308e166c9a3b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:11.778611643 +0000 UTC m=+4.853066993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rvfx" (UID: "5557e025-30b7-4315-b7a1-308e166c9a3b") : secret "insights-runtime-extractor-tls" not found Oct 30 19:04:09.778972 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:09.778681 5047 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:04:09.778972 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:09.778691 5047 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4d8bb96d-62jtz: secret "image-registry-tls" not found Oct 30 19:04:09.778972 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:09.778729 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls podName:6af08cb7-fa60-491b-88fd-07c1ace45779 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:11.778717217 +0000 UTC m=+4.853172554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls") pod "image-registry-5d4d8bb96d-62jtz" (UID: "6af08cb7-fa60-491b-88fd-07c1ace45779") : secret "image-registry-tls" not found Oct 30 19:04:09.778972 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:09.778769 5047 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:04:09.778972 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:09.778775 5047 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd9cf966f-mn64l: secret "image-registry-tls" not found Oct 30 19:04:09.778972 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:09.778794 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls podName:3ac3ab69-eead-4eca-af11-53fec376a6ac nodeName:}" failed. No retries permitted until 2026-10-30 19:04:11.778787998 +0000 UTC m=+4.853243335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls") pod "image-registry-5fd9cf966f-mn64l" (UID: "3ac3ab69-eead-4eca-af11-53fec376a6ac") : secret "image-registry-tls" not found Oct 30 19:04:09.778972 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:09.778831 5047 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:04:09.778972 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:09.778850 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs podName:93f0a70e-357e-4d65-b554-42363280bebd nodeName:}" failed. No retries permitted until 2026-10-30 19:04:11.778844736 +0000 UTC m=+4.853300072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs") pod "network-metrics-daemon-frj4v" (UID: "93f0a70e-357e-4d65-b554-42363280bebd") : secret "metrics-daemon-secret" not found Oct 30 19:04:09.778972 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:09.778890 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle podName:20403b1e-e479-45b9-bb85-52cf10725d86 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:11.778884589 +0000 UTC m=+4.853339925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle") pod "router-default-77d478f789-d787c" (UID: "20403b1e-e479-45b9-bb85-52cf10725d86") : configmap references non-existent config key: service-ca.crt Oct 30 19:04:09.778972 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:09.778934 5047 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:04:09.778972 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:09.778964 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls podName:b50175d2-e6bb-463a-9602-5bf9e4719a15 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:11.778955048 +0000 UTC m=+4.853410388 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lx6v2" (UID: "b50175d2-e6bb-463a-9602-5bf9e4719a15") : secret "samples-operator-tls" not found Oct 30 19:04:09.872694 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:09.872633 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ccd7w_3e626b03-28b0-4840-bbe2-518834311815/serve-healthcheck-canary/0.log" Oct 30 19:04:10.071333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:10.071199 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-6zk7f_452b3edc-e035-446c-9b7b-ee8154f3c6c3/migrator/0.log" Oct 30 19:04:10.120858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:10.120827 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jqrww" event={"ID":"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b","Type":"ContainerStarted","Data":"7d305de0af43dbecaf4384c4ceb75a4fd3762f16efb8215270e2c2c1ae252339"} Oct 30 19:04:10.123023 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:10.122979 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:04:10.125048 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:10.124912 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-nxngq" event={"ID":"bcf8833f-ed61-40f4-880c-2645ffae454b","Type":"ContainerStarted","Data":"876a5d86268c91dd0d122c5148813d696cbdd0197666678630fc99db00620588"} Oct 30 19:04:10.270584 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:10.270552 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-lp2xk_facfc926-9288-41c4-bdc3-5d8acf446292/migrator/0.log" Oct 30 19:04:10.436152 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:10.436124 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:04:10.436532 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:10.436515 5047 scope.go:122] "RemoveContainer" containerID="673c43d635dd91bc96e71ed74ed420916c8405959101bc1bfdb3202aac48722b" Oct 30 19:04:10.436727 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:10.436700 5047 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6h7t5_openshift-console-operator(bd7cda5e-1a4b-45f8-9cab-f96111986c08)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" podUID="bd7cda5e-1a4b-45f8-9cab-f96111986c08" Oct 30 19:04:10.472381 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:10.472234 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l_721cf657-76b1-4fdf-868d-d84cb9fa0b14/kube-storage-version-migrator-operator/0.log" Oct 30 19:04:11.133644 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.133612 5047 generic.go:356] "Generic (PLEG): container finished" podID="721cf657-76b1-4fdf-868d-d84cb9fa0b14" containerID="1bbf3f90a4c1ebd958b962674daf83104910396b0d3cf3e4678b3fb830e86bd6" exitCode=0 Oct 30 19:04:11.134205 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.133670 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" event={"ID":"721cf657-76b1-4fdf-868d-d84cb9fa0b14","Type":"ContainerDied","Data":"1bbf3f90a4c1ebd958b962674daf83104910396b0d3cf3e4678b3fb830e86bd6"} Oct 30 19:04:11.134205 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.134111 5047 scope.go:122] "RemoveContainer" containerID="1bbf3f90a4c1ebd958b962674daf83104910396b0d3cf3e4678b3fb830e86bd6" Oct 30 19:04:11.137582 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.137553 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jqrww" event={"ID":"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b","Type":"ContainerStarted","Data":"fd14f36bf938ee9d0278e24ee305da4c184e2f7968e336d0eb0dc0615b03d6bf"} Oct 30 19:04:11.140772 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.140744 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x4r6k" event={"ID":"540944ec-348d-46e1-836b-9aeb07d16713","Type":"ContainerStarted","Data":"d304359cfcb14a9434d67a94d1e4081f28ada9bc515889b38d7bc06c13182a3b"} Oct 30 19:04:11.144856 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.144822 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" event={"ID":"c6188415-c8d5-4972-9d6d-f6e3bcf4b079","Type":"ContainerDied","Data":"156210b86be7e2ecd7c970bad403ffa89797701f1905637ce60f467218bb38d7"} Oct 30 19:04:11.145290 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.145273 5047 scope.go:122] "RemoveContainer" containerID="156210b86be7e2ecd7c970bad403ffa89797701f1905637ce60f467218bb38d7" Oct 30 19:04:11.145938 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.144787 5047 generic.go:356] "Generic (PLEG): container finished" podID="c6188415-c8d5-4972-9d6d-f6e3bcf4b079" containerID="156210b86be7e2ecd7c970bad403ffa89797701f1905637ce60f467218bb38d7" exitCode=0 Oct 30 19:04:11.154476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.154446 5047 generic.go:356] "Generic (PLEG): container finished" podID="11d31a53-ae1b-4d46-aa8c-6c00e89ed539" containerID="81c26b6e909fc8f6eedbfe63cb64a1fe1e7a5f03c1e96645094aaf67c2c245d0" exitCode=0 Oct 30 19:04:11.154633 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.154611 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wd8lm" event={"ID":"11d31a53-ae1b-4d46-aa8c-6c00e89ed539","Type":"ContainerDied","Data":"81c26b6e909fc8f6eedbfe63cb64a1fe1e7a5f03c1e96645094aaf67c2c245d0"} Oct 30 19:04:11.155072 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.155054 5047 scope.go:122] "RemoveContainer" containerID="81c26b6e909fc8f6eedbfe63cb64a1fe1e7a5f03c1e96645094aaf67c2c245d0" Oct 30 19:04:11.455794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.455759 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d26nk" Oct 30 19:04:11.796599 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.796520 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:04:11.796599 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.796581 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:11.796806 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.796621 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:11.796806 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.796648 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:11.796806 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.796670 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:04:11.796806 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.796691 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:11.796806 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:11.796738 5047 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:04:11.797065 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:11.796823 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls podName:5557e025-30b7-4315-b7a1-308e166c9a3b nodeName:}" failed. No retries permitted until 2026-10-30 19:04:15.796803557 +0000 UTC m=+8.871258894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5rvfx" (UID: "5557e025-30b7-4315-b7a1-308e166c9a3b") : secret "insights-runtime-extractor-tls" not found Oct 30 19:04:11.797477 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.797453 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20403b1e-e479-45b9-bb85-52cf10725d86-service-ca-bundle\") pod \"router-default-77d478f789-d787c\" (UID: \"20403b1e-e479-45b9-bb85-52cf10725d86\") " pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:11.799353 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.799328 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls\") pod \"image-registry-5d4d8bb96d-62jtz\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:11.799443 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.799378 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f0a70e-357e-4d65-b554-42363280bebd-metrics-certs\") pod \"network-metrics-daemon-frj4v\" (UID: \"93f0a70e-357e-4d65-b554-42363280bebd\") " pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:04:11.799615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.799589 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b50175d2-e6bb-463a-9602-5bf9e4719a15-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lx6v2\" (UID: \"b50175d2-e6bb-463a-9602-5bf9e4719a15\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:04:11.799676 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.799589 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls\") pod \"image-registry-5fd9cf966f-mn64l\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:11.938977 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.938940 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:11.939155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.939035 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" Oct 30 19:04:11.940022 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.939983 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:11.943676 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.943651 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-frj4v" Oct 30 19:04:11.950491 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:11.950466 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:12.160245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:12.160212 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9ls9s" event={"ID":"c6188415-c8d5-4972-9d6d-f6e3bcf4b079","Type":"ContainerStarted","Data":"485866db7d812ff378806d9b2a8528e234edec534811e5c2c02c02972a8b4103"} Oct 30 19:04:12.162147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:12.162121 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wd8lm" event={"ID":"11d31a53-ae1b-4d46-aa8c-6c00e89ed539","Type":"ContainerStarted","Data":"fb46dd0740e798d0d2b1e9af3a066bcaac9f3ea18a3e27ccb4194c38d60deb1c"} Oct 30 19:04:12.163884 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:12.163856 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l" event={"ID":"721cf657-76b1-4fdf-868d-d84cb9fa0b14","Type":"ContainerStarted","Data":"71f7a1c105baf1bfa559aba9d21a184f20422999367e258910e6f6f6e84886f3"} Oct 30 19:04:12.165639 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:12.165616 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jqrww" event={"ID":"a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b","Type":"ContainerStarted","Data":"035f18437f1586c50d4e978ea260f76ec13ad842430738596a949cd00afc6242"} Oct 30 19:04:12.267422 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:12.267391 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2"] Oct 30 19:04:12.271326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:12.271304 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d4d8bb96d-62jtz"] Oct 30 19:04:12.274141 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:12.274116 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af08cb7_fa60_491b_88fd_07c1ace45779.slice/crio-aeb76474679e2315996082b5a929ded7b85e95c062f9c1b195847d89b04ba25b WatchSource:0}: Error finding container aeb76474679e2315996082b5a929ded7b85e95c062f9c1b195847d89b04ba25b: Status 404 returned error can't find the container with id aeb76474679e2315996082b5a929ded7b85e95c062f9c1b195847d89b04ba25b Oct 30 19:04:12.282092 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:12.281654 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-frj4v"] Oct 30 19:04:12.285669 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:12.285645 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77d478f789-d787c"] Oct 30 19:04:12.287108 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:12.286780 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f0a70e_357e_4d65_b554_42363280bebd.slice/crio-b09b5ba70ffdfa4dd0349dbb7219f769185217dc8e651cce5cde32f3783023ac WatchSource:0}: Error finding container b09b5ba70ffdfa4dd0349dbb7219f769185217dc8e651cce5cde32f3783023ac: Status 404 returned error can't find the container with id b09b5ba70ffdfa4dd0349dbb7219f769185217dc8e651cce5cde32f3783023ac Oct 30 19:04:12.289645 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:12.289622 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20403b1e_e479_45b9_bb85_52cf10725d86.slice/crio-2d99d4b8f7bae8c81fbd96c1de37832cf3272aacbbe7a4c6d078e0fbc2b65d3c WatchSource:0}: Error finding container 2d99d4b8f7bae8c81fbd96c1de37832cf3272aacbbe7a4c6d078e0fbc2b65d3c: Status 404 returned error can't find the container with id 2d99d4b8f7bae8c81fbd96c1de37832cf3272aacbbe7a4c6d078e0fbc2b65d3c Oct 30 19:04:12.301689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:12.301661 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fd9cf966f-mn64l"] Oct 30 19:04:13.172893 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:13.172805 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" event={"ID":"b50175d2-e6bb-463a-9602-5bf9e4719a15","Type":"ContainerStarted","Data":"cdf45714daa1db260dc64b4567fe5fc08e0fee53f54e5493ce2fd463be479f35"} Oct 30 19:04:13.174565 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:13.174536 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77d478f789-d787c" event={"ID":"20403b1e-e479-45b9-bb85-52cf10725d86","Type":"ContainerStarted","Data":"64a1886a3563f03e3de24b54ed0a1ba0e1b1903f70902d5dd8b57cf34df48cfe"} Oct 30 19:04:13.174665 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:13.174591 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77d478f789-d787c" event={"ID":"20403b1e-e479-45b9-bb85-52cf10725d86","Type":"ContainerStarted","Data":"2d99d4b8f7bae8c81fbd96c1de37832cf3272aacbbe7a4c6d078e0fbc2b65d3c"} Oct 30 19:04:13.176263 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:13.176223 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" event={"ID":"6af08cb7-fa60-491b-88fd-07c1ace45779","Type":"ContainerStarted","Data":"9a1dbd58dbaa05e08534f529ad3df6a43830307d4164e500c69428358a33ec84"} Oct 30 19:04:13.176363 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:13.176271 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" event={"ID":"6af08cb7-fa60-491b-88fd-07c1ace45779","Type":"ContainerStarted","Data":"aeb76474679e2315996082b5a929ded7b85e95c062f9c1b195847d89b04ba25b"} Oct 30 19:04:13.178102 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:13.177994 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" event={"ID":"3ac3ab69-eead-4eca-af11-53fec376a6ac","Type":"ContainerStarted","Data":"3786bb68ae3fc3d14d041fec47a72057f86df44d346808845bedb865e8853283"} Oct 30 19:04:13.178191 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:13.178122 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" event={"ID":"3ac3ab69-eead-4eca-af11-53fec376a6ac","Type":"ContainerStarted","Data":"7a7329e3cf0cb878f0f65ca567d9e2189bd568474405f7975fca6e9deff8b993"} Oct 30 19:04:13.179730 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:13.179712 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-frj4v" event={"ID":"93f0a70e-357e-4d65-b554-42363280bebd","Type":"ContainerStarted","Data":"b09b5ba70ffdfa4dd0349dbb7219f769185217dc8e651cce5cde32f3783023ac"} Oct 30 19:04:13.179811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:13.179748 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jqrww" Oct 30 19:04:13.236473 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:13.236448 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:13.236672 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:13.236653 5047 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:04:13.939560 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:13.939523 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:13.941990 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:13.941969 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:14.186429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:14.186389 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-frj4v" event={"ID":"93f0a70e-357e-4d65-b554-42363280bebd","Type":"ContainerStarted","Data":"3fa47355e8e54a4caab873d387de52b82fcc54b60fce39040807dad1dc48d942"} Oct 30 19:04:14.186882 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:14.186458 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-frj4v" event={"ID":"93f0a70e-357e-4d65-b554-42363280bebd","Type":"ContainerStarted","Data":"27df6ce02b047461963434d8f914a703ae11fee2cdd45c11c6272e38e6784676"} Oct 30 19:04:14.188276 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:14.188246 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" event={"ID":"b50175d2-e6bb-463a-9602-5bf9e4719a15","Type":"ContainerStarted","Data":"eac4dbaeb342168452e099a5fbb56c0275a78a9b3a1f562a7593fb92bbe81a10"} Oct 30 19:04:14.188395 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:14.188293 5047 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:04:14.188395 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:14.188293 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lx6v2" event={"ID":"b50175d2-e6bb-463a-9602-5bf9e4719a15","Type":"ContainerStarted","Data":"f10b4046bb5bc74f10b6fba2a13217124f71c73d7999f275ba40595568a1a027"} Oct 30 19:04:14.188635 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:14.188617 5047 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:04:14.188873 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:14.188840 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:14.189702 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:14.189654 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-77d478f789-d787c" Oct 30 19:04:14.991881 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:14.991836 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:15.836728 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:15.836685 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:15.838944 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:15.838923 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5557e025-30b7-4315-b7a1-308e166c9a3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5rvfx\" (UID: \"5557e025-30b7-4315-b7a1-308e166c9a3b\") " pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:15.854948 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:15.854920 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5rvfx" Oct 30 19:04:16.141403 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:16.141378 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5rvfx"] Oct 30 19:04:16.143818 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:16.143790 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5557e025_30b7_4315_b7a1_308e166c9a3b.slice/crio-53d4502301b3e640ba016b6a584456a3558534369937c1bdf526794b46f3454e WatchSource:0}: Error finding container 53d4502301b3e640ba016b6a584456a3558534369937c1bdf526794b46f3454e: Status 404 returned error can't find the container with id 53d4502301b3e640ba016b6a584456a3558534369937c1bdf526794b46f3454e Oct 30 19:04:16.198593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:16.198526 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5rvfx" event={"ID":"5557e025-30b7-4315-b7a1-308e166c9a3b","Type":"ContainerStarted","Data":"53d4502301b3e640ba016b6a584456a3558534369937c1bdf526794b46f3454e"} Oct 30 19:04:16.877674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:16.877647 5047 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:04:16.878039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:16.878025 5047 scope.go:122] "RemoveContainer" containerID="673c43d635dd91bc96e71ed74ed420916c8405959101bc1bfdb3202aac48722b" Oct 30 19:04:16.878221 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:16.878204 5047 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6h7t5_openshift-console-operator(bd7cda5e-1a4b-45f8-9cab-f96111986c08)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" podUID="bd7cda5e-1a4b-45f8-9cab-f96111986c08" Oct 30 19:04:17.203845 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:17.203774 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5rvfx" event={"ID":"5557e025-30b7-4315-b7a1-308e166c9a3b","Type":"ContainerStarted","Data":"2be1db945f9d059eb27a77fab54e9169a8046b4db13415eb75f2109bcdbcc887"} Oct 30 19:04:17.203845 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:17.203829 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5rvfx" event={"ID":"5557e025-30b7-4315-b7a1-308e166c9a3b","Type":"ContainerStarted","Data":"7fdff1152503dacbb86b5a77c74e8e3e51b62826e18da325fb9e8d3c6718abe8"} Oct 30 19:04:18.135409 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:18.135368 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:19.041435 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:19.041405 5047 scope.go:122] "RemoveContainer" containerID="673c43d635dd91bc96e71ed74ed420916c8405959101bc1bfdb3202aac48722b" Oct 30 19:04:19.213987 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:19.213950 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5rvfx" event={"ID":"5557e025-30b7-4315-b7a1-308e166c9a3b","Type":"ContainerStarted","Data":"4ee6f38d0ca50e4fd5cab899a6f1ff6fdc2b9df61c6e4ba5b10849de30a91826"} Oct 30 19:04:19.215902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:19.215882 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:04:19.216016 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:19.215935 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" event={"ID":"bd7cda5e-1a4b-45f8-9cab-f96111986c08","Type":"ContainerStarted","Data":"f1278fd11009df7f8c9eb74277eecdd55c2574a6e53d70db07c72d7f4b168125"} Oct 30 19:04:19.216217 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:19.216199 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:04:19.217326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:19.217303 5047 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-6h7t5 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body= Oct 30 19:04:19.217393 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:19.217359 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" podUID="bd7cda5e-1a4b-45f8-9cab-f96111986c08" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" Oct 30 19:04:20.225133 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:20.225105 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6h7t5" Oct 30 19:04:21.971635 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:21.971607 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qftp" Oct 30 19:04:23.190266 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:23.190238 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jqrww" Oct 30 19:04:23.837753 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:23.837719 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-m5mwl-99mnf"] Oct 30 19:04:23.840949 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:23.840921 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-m5mwl-99mnf" Oct 30 19:04:23.875183 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:23.875137 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-m5mwl-99mnf"] Oct 30 19:04:23.909851 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:23.909817 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d4d8bb96d-62jtz"] Oct 30 19:04:24.010930 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:24.010899 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxvxm\" (UniqueName: \"kubernetes.io/projected/9c46dbf4-49e2-4435-8362-255dcd09ebbf-kube-api-access-cxvxm\") pod \"periodic-gathering-m5mwl-99mnf\" (UID: \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\") " pod="openshift-insights/periodic-gathering-m5mwl-99mnf" Oct 30 19:04:24.011109 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:24.010957 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9c46dbf4-49e2-4435-8362-255dcd09ebbf-archives-path\") pod \"periodic-gathering-m5mwl-99mnf\" (UID: \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\") " pod="openshift-insights/periodic-gathering-m5mwl-99mnf" Oct 30 19:04:24.011109 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:24.011068 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46dbf4-49e2-4435-8362-255dcd09ebbf-service-ca-bundle\") pod \"periodic-gathering-m5mwl-99mnf\" (UID: \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\") " pod="openshift-insights/periodic-gathering-m5mwl-99mnf" Oct 30 19:04:24.011195 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:24.011121 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46dbf4-49e2-4435-8362-255dcd09ebbf-trusted-ca-bundle\") pod \"periodic-gathering-m5mwl-99mnf\" (UID: \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\") " pod="openshift-insights/periodic-gathering-m5mwl-99mnf" Oct 30 19:04:24.112500 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:24.112422 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9c46dbf4-49e2-4435-8362-255dcd09ebbf-archives-path\") pod \"periodic-gathering-m5mwl-99mnf\" (UID: \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\") " pod="openshift-insights/periodic-gathering-m5mwl-99mnf" Oct 30 19:04:24.112500 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:24.112457 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46dbf4-49e2-4435-8362-255dcd09ebbf-service-ca-bundle\") pod \"periodic-gathering-m5mwl-99mnf\" (UID: \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\") " pod="openshift-insights/periodic-gathering-m5mwl-99mnf" Oct 30 19:04:24.112500 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:24.112482 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46dbf4-49e2-4435-8362-255dcd09ebbf-trusted-ca-bundle\") pod \"periodic-gathering-m5mwl-99mnf\" (UID: \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\") " pod="openshift-insights/periodic-gathering-m5mwl-99mnf" Oct 30 19:04:24.112798 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:24.112522 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxvxm\" (UniqueName: \"kubernetes.io/projected/9c46dbf4-49e2-4435-8362-255dcd09ebbf-kube-api-access-cxvxm\") pod \"periodic-gathering-m5mwl-99mnf\" (UID: \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\") " pod="openshift-insights/periodic-gathering-m5mwl-99mnf" Oct 30 19:04:24.112864 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:24.112806 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9c46dbf4-49e2-4435-8362-255dcd09ebbf-archives-path\") pod \"periodic-gathering-m5mwl-99mnf\" (UID: \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\") " pod="openshift-insights/periodic-gathering-m5mwl-99mnf" Oct 30 19:04:24.113210 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:24.113184 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46dbf4-49e2-4435-8362-255dcd09ebbf-service-ca-bundle\") pod \"periodic-gathering-m5mwl-99mnf\" (UID: \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\") " pod="openshift-insights/periodic-gathering-m5mwl-99mnf" Oct 30 19:04:24.113317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:24.113298 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46dbf4-49e2-4435-8362-255dcd09ebbf-trusted-ca-bundle\") pod \"periodic-gathering-m5mwl-99mnf\" (UID: \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\") " pod="openshift-insights/periodic-gathering-m5mwl-99mnf" Oct 30 19:04:24.131250 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:24.131224 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxvxm\" (UniqueName: \"kubernetes.io/projected/9c46dbf4-49e2-4435-8362-255dcd09ebbf-kube-api-access-cxvxm\") pod \"periodic-gathering-m5mwl-99mnf\" (UID: \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\") " pod="openshift-insights/periodic-gathering-m5mwl-99mnf" Oct 30 19:04:24.151968 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:24.151946 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-m5mwl-99mnf" Oct 30 19:04:24.440819 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:24.440786 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-m5mwl-99mnf"] Oct 30 19:04:24.443552 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:24.443521 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c46dbf4_49e2_4435_8362_255dcd09ebbf.slice/crio-bff0398fb79c0c472cfe3282517100f2415acf9733bda4923750aaa5c1dd6172 WatchSource:0}: Error finding container bff0398fb79c0c472cfe3282517100f2415acf9733bda4923750aaa5c1dd6172: Status 404 returned error can't find the container with id bff0398fb79c0c472cfe3282517100f2415acf9733bda4923750aaa5c1dd6172 Oct 30 19:04:25.245165 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:25.245128 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-m5mwl-99mnf" event={"ID":"9c46dbf4-49e2-4435-8362-255dcd09ebbf","Type":"ContainerStarted","Data":"e22993b6753f3d7ad3c27b5439c08e60c221206b1b8234589191bfa9efc840fd"} Oct 30 19:04:25.245351 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:25.245184 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-m5mwl-99mnf" event={"ID":"9c46dbf4-49e2-4435-8362-255dcd09ebbf","Type":"ContainerStarted","Data":"bff0398fb79c0c472cfe3282517100f2415acf9733bda4923750aaa5c1dd6172"} Oct 30 19:04:25.280354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:25.280297 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-m5mwl-99mnf" podStartSLOduration=2.2802812550000002 podStartE2EDuration="2.280281255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:25.278581119 +0000 UTC m=+18.353036478" watchObservedRunningTime="2026-10-30 19:04:25.280281255 +0000 UTC m=+18.354736607" Oct 30 19:04:26.280871 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:26.280839 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jqrww_a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b/dns/0.log" Oct 30 19:04:26.481285 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:26.481253 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jqrww_a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b/kube-rbac-proxy/0.log" Oct 30 19:04:27.480206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:27.480175 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xpxgd_415a1d40-3275-4624-98cd-3df86bbc819a/dns-node-resolver/0.log" Oct 30 19:04:27.681432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:27.681384 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d4d8bb96d-62jtz_6af08cb7-fa60-491b-88fd-07c1ace45779/registry/0.log" Oct 30 19:04:27.881231 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:27.881199 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5fd9cf966f-mn64l_3ac3ab69-eead-4eca-af11-53fec376a6ac/registry/0.log" Oct 30 19:04:28.140607 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:28.140519 5047 patch_prober.go:29] interesting pod/image-registry-5fd9cf966f-mn64l container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 19:04:28.140607 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:28.140580 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" podUID="3ac3ab69-eead-4eca-af11-53fec376a6ac" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:04:28.480978 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:28.480892 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tbttr_830ddb4e-a8f8-406c-9e89-755bb0d48e0e/node-ca/0.log" Oct 30 19:04:28.880859 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:28.880829 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77d478f789-d787c_20403b1e-e479-45b9-bb85-52cf10725d86/router/0.log" Oct 30 19:04:29.272686 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:29.272608 5047 patch_prober.go:29] interesting pod/image-registry-5fd9cf966f-mn64l container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 19:04:29.272686 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:29.272660 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" podUID="3ac3ab69-eead-4eca-af11-53fec376a6ac" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:04:29.280209 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:29.280143 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ccd7w_3e626b03-28b0-4840-bbe2-518834311815/serve-healthcheck-canary/0.log" Oct 30 19:04:31.253825 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.253740 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xbvmn"] Oct 30 19:04:31.286406 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.286379 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.326321 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.326252 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:04:31.326321 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.326259 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:04:31.326520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.326393 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sz8sw\"" Oct 30 19:04:31.362564 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.362529 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:04:31.362937 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.362915 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:04:31.371455 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.371427 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33d88d43-afaf-4899-af60-57a0020af5f1-node-exporter-textfile\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.371579 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.371463 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcgwc\" (UniqueName: \"kubernetes.io/projected/33d88d43-afaf-4899-af60-57a0020af5f1-kube-api-access-qcgwc\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.371579 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.371483 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33d88d43-afaf-4899-af60-57a0020af5f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.371579 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.371563 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33d88d43-afaf-4899-af60-57a0020af5f1-node-exporter-wtmp\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.371739 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.371719 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33d88d43-afaf-4899-af60-57a0020af5f1-node-exporter-tls\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.371775 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.371756 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33d88d43-afaf-4899-af60-57a0020af5f1-root\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.371814 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.371785 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33d88d43-afaf-4899-af60-57a0020af5f1-metrics-client-ca\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.371874 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.371853 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33d88d43-afaf-4899-af60-57a0020af5f1-sys\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.371969 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.371955 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33d88d43-afaf-4899-af60-57a0020af5f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.472576 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.472537 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33d88d43-afaf-4899-af60-57a0020af5f1-node-exporter-wtmp\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.472764 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.472586 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33d88d43-afaf-4899-af60-57a0020af5f1-node-exporter-tls\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.472764 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.472614 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33d88d43-afaf-4899-af60-57a0020af5f1-root\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.472764 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.472665 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33d88d43-afaf-4899-af60-57a0020af5f1-root\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.472764 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.472725 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33d88d43-afaf-4899-af60-57a0020af5f1-metrics-client-ca\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.472970 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.472758 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33d88d43-afaf-4899-af60-57a0020af5f1-node-exporter-wtmp\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.472970 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.472805 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33d88d43-afaf-4899-af60-57a0020af5f1-sys\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.472970 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.472838 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33d88d43-afaf-4899-af60-57a0020af5f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.472970 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.472873 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33d88d43-afaf-4899-af60-57a0020af5f1-node-exporter-textfile\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.472970 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.472897 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33d88d43-afaf-4899-af60-57a0020af5f1-sys\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.472970 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.472914 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcgwc\" (UniqueName: \"kubernetes.io/projected/33d88d43-afaf-4899-af60-57a0020af5f1-kube-api-access-qcgwc\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.472970 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.472940 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33d88d43-afaf-4899-af60-57a0020af5f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.473333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.473262 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33d88d43-afaf-4899-af60-57a0020af5f1-node-exporter-textfile\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.473444 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.473424 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33d88d43-afaf-4899-af60-57a0020af5f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.473479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.473424 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33d88d43-afaf-4899-af60-57a0020af5f1-metrics-client-ca\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.475057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.475027 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33d88d43-afaf-4899-af60-57a0020af5f1-node-exporter-tls\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.475160 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.475139 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33d88d43-afaf-4899-af60-57a0020af5f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.492444 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.492405 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcgwc\" (UniqueName: \"kubernetes.io/projected/33d88d43-afaf-4899-af60-57a0020af5f1-kube-api-access-qcgwc\") pod \"node-exporter-xbvmn\" (UID: \"33d88d43-afaf-4899-af60-57a0020af5f1\") " pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.658111 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:31.658076 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xbvmn" Oct 30 19:04:31.698161 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:04:31.698130 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d88d43_afaf_4899_af60_57a0020af5f1.slice/crio-3044cc8edaa98b9c892a617f49630a3cb25a6654270866cbe82723dd76a40d1b WatchSource:0}: Error finding container 3044cc8edaa98b9c892a617f49630a3cb25a6654270866cbe82723dd76a40d1b: Status 404 returned error can't find the container with id 3044cc8edaa98b9c892a617f49630a3cb25a6654270866cbe82723dd76a40d1b Oct 30 19:04:32.285461 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:32.285420 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xbvmn" event={"ID":"33d88d43-afaf-4899-af60-57a0020af5f1","Type":"ContainerStarted","Data":"3044cc8edaa98b9c892a617f49630a3cb25a6654270866cbe82723dd76a40d1b"} Oct 30 19:04:33.291296 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:33.291262 5047 generic.go:356] "Generic (PLEG): container finished" podID="33d88d43-afaf-4899-af60-57a0020af5f1" containerID="e4b7ba8030e6f85d76caf013124a9f356e0ecd6c158ab7cdc9c703667e67b9a0" exitCode=0 Oct 30 19:04:33.291694 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:33.291348 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xbvmn" event={"ID":"33d88d43-afaf-4899-af60-57a0020af5f1","Type":"ContainerDied","Data":"e4b7ba8030e6f85d76caf013124a9f356e0ecd6c158ab7cdc9c703667e67b9a0"} Oct 30 19:04:33.915778 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:33.915747 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:34.299222 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:34.299149 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xbvmn" event={"ID":"33d88d43-afaf-4899-af60-57a0020af5f1","Type":"ContainerStarted","Data":"dcf585a5a410f9a412c177917f68cb8bf3f9ad2ae01c901e46de5edd72099f2c"} Oct 30 19:04:34.299222 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:34.299211 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xbvmn" event={"ID":"33d88d43-afaf-4899-af60-57a0020af5f1","Type":"ContainerStarted","Data":"e9785588b48d85ba6d5a9a179aea78f6e55fe7c5ffc68191d4f7543581714e22"} Oct 30 19:04:34.335477 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:34.335429 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xbvmn" podStartSLOduration=1.929054104 podStartE2EDuration="3.335415351s" totalImagesPullingTime="1.354149241s" totalInitContainerRuntime="52.212006ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:34.334716567 +0000 UTC m=+27.409171942" watchObservedRunningTime="2026-10-30 19:04:34.335415351 +0000 UTC m=+27.409870710" Oct 30 19:04:38.140581 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:38.140544 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:04:39.117466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:39.117435 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zwhqt" Oct 30 19:04:39.599810 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:39.599771 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:04:46.270754 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:46.270717 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5fd9cf966f-mn64l"] Oct 30 19:04:48.930602 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:48.930492 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" podUID="6af08cb7-fa60-491b-88fd-07c1ace45779" containerName="registry" containerID="cri-o://9a1dbd58dbaa05e08534f529ad3df6a43830307d4164e500c69428358a33ec84" gracePeriod=30 Oct 30 19:04:49.183194 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.183137 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:49.242301 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.242271 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-certificates\") pod \"6af08cb7-fa60-491b-88fd-07c1ace45779\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " Oct 30 19:04:49.242301 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.242311 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-trusted-ca\") pod \"6af08cb7-fa60-491b-88fd-07c1ace45779\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " Oct 30 19:04:49.242585 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.242364 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-installation-pull-secrets\") pod \"6af08cb7-fa60-491b-88fd-07c1ace45779\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " Oct 30 19:04:49.242585 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.242403 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af08cb7-fa60-491b-88fd-07c1ace45779-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6af08cb7-fa60-491b-88fd-07c1ace45779-ca-trust-extracted\") pod \"6af08cb7-fa60-491b-88fd-07c1ace45779\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " Oct 30 19:04:49.242585 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.242445 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-bound-sa-token\") pod \"6af08cb7-fa60-491b-88fd-07c1ace45779\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " Oct 30 19:04:49.242585 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.242474 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls\") pod \"6af08cb7-fa60-491b-88fd-07c1ace45779\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " Oct 30 19:04:49.242585 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.242503 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-kube-api-access-sp2w6\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-kube-api-access-sp2w6\") pod \"6af08cb7-fa60-491b-88fd-07c1ace45779\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " Oct 30 19:04:49.242585 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.242561 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-image-registry-private-configuration\") pod \"6af08cb7-fa60-491b-88fd-07c1ace45779\" (UID: \"6af08cb7-fa60-491b-88fd-07c1ace45779\") " Oct 30 19:04:49.242847 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.242749 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-certificates" pod "6af08cb7-fa60-491b-88fd-07c1ace45779" (UID: "6af08cb7-fa60-491b-88fd-07c1ace45779"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:49.242886 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.242865 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-trusted-ca" pod "6af08cb7-fa60-491b-88fd-07c1ace45779" (UID: "6af08cb7-fa60-491b-88fd-07c1ace45779"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:49.245682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.245649 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-installation-pull-secrets" pod "6af08cb7-fa60-491b-88fd-07c1ace45779" (UID: "6af08cb7-fa60-491b-88fd-07c1ace45779"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:49.245811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.245754 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-image-registry-private-configuration" pod "6af08cb7-fa60-491b-88fd-07c1ace45779" (UID: "6af08cb7-fa60-491b-88fd-07c1ace45779"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:49.245870 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.245833 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls" pod "6af08cb7-fa60-491b-88fd-07c1ace45779" (UID: "6af08cb7-fa60-491b-88fd-07c1ace45779"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:49.245908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.245876 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-kube-api-access-sp2w6" pod "6af08cb7-fa60-491b-88fd-07c1ace45779" (UID: "6af08cb7-fa60-491b-88fd-07c1ace45779"). InnerVolumeSpecName "kube-api-access-sp2w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:49.245945 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.245910 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-bound-sa-token" pod "6af08cb7-fa60-491b-88fd-07c1ace45779" (UID: "6af08cb7-fa60-491b-88fd-07c1ace45779"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:49.249728 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.249702 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af08cb7-fa60-491b-88fd-07c1ace45779-ca-trust-extracted" pod "6af08cb7-fa60-491b-88fd-07c1ace45779" (UID: "6af08cb7-fa60-491b-88fd-07c1ace45779"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:49.343862 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.343827 5047 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-image-registry-private-configuration\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:04:49.343862 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.343859 5047 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-certificates\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:04:49.343862 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.343870 5047 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6af08cb7-fa60-491b-88fd-07c1ace45779-trusted-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:04:49.344098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.343880 5047 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6af08cb7-fa60-491b-88fd-07c1ace45779-installation-pull-secrets\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:04:49.344098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.343889 5047 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6af08cb7-fa60-491b-88fd-07c1ace45779-ca-trust-extracted\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:04:49.344098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.343898 5047 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-bound-sa-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:04:49.344098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.343907 5047 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-registry-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:04:49.344098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.343915 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp2w6\" (UniqueName: \"kubernetes.io/projected/6af08cb7-fa60-491b-88fd-07c1ace45779-kube-api-access-sp2w6\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:04:49.366185 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.366153 5047 generic.go:356] "Generic (PLEG): container finished" podID="6af08cb7-fa60-491b-88fd-07c1ace45779" containerID="9a1dbd58dbaa05e08534f529ad3df6a43830307d4164e500c69428358a33ec84" exitCode=0 Oct 30 19:04:49.366351 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.366195 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" Oct 30 19:04:49.366351 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.366243 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" event={"ID":"6af08cb7-fa60-491b-88fd-07c1ace45779","Type":"ContainerDied","Data":"9a1dbd58dbaa05e08534f529ad3df6a43830307d4164e500c69428358a33ec84"} Oct 30 19:04:49.366351 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.366305 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d4d8bb96d-62jtz" event={"ID":"6af08cb7-fa60-491b-88fd-07c1ace45779","Type":"ContainerDied","Data":"aeb76474679e2315996082b5a929ded7b85e95c062f9c1b195847d89b04ba25b"} Oct 30 19:04:49.366351 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.366330 5047 scope.go:122] "RemoveContainer" containerID="9a1dbd58dbaa05e08534f529ad3df6a43830307d4164e500c69428358a33ec84" Oct 30 19:04:49.376026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.375987 5047 scope.go:122] "RemoveContainer" containerID="9a1dbd58dbaa05e08534f529ad3df6a43830307d4164e500c69428358a33ec84" Oct 30 19:04:49.376315 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:49.376296 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1dbd58dbaa05e08534f529ad3df6a43830307d4164e500c69428358a33ec84\": container with ID starting with 9a1dbd58dbaa05e08534f529ad3df6a43830307d4164e500c69428358a33ec84 not found: ID does not exist" containerID="9a1dbd58dbaa05e08534f529ad3df6a43830307d4164e500c69428358a33ec84" Oct 30 19:04:49.376372 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.376326 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1dbd58dbaa05e08534f529ad3df6a43830307d4164e500c69428358a33ec84"} err="failed to get container status \"9a1dbd58dbaa05e08534f529ad3df6a43830307d4164e500c69428358a33ec84\": rpc error: code = NotFound desc = could not find container \"9a1dbd58dbaa05e08534f529ad3df6a43830307d4164e500c69428358a33ec84\": container with ID starting with 9a1dbd58dbaa05e08534f529ad3df6a43830307d4164e500c69428358a33ec84 not found: ID does not exist" Oct 30 19:04:49.409523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.409496 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d4d8bb96d-62jtz"] Oct 30 19:04:49.414219 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:49.414197 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d4d8bb96d-62jtz"] Oct 30 19:04:51.046279 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:51.046237 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af08cb7-fa60-491b-88fd-07c1ace45779" path="/var/lib/kubelet/pods/6af08cb7-fa60-491b-88fd-07c1ace45779/volumes" Oct 30 19:04:53.020279 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:53.020245 5047 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d26nk_633f6b63-a067-470b-bec9-e913e7f28589/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:56.085944 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.085906 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hkjk5-l9rq7"] Oct 30 19:04:56.086372 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.086166 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" podUID="1a7defa5-7e66-4e17-8dc8-e74137e4f054" containerName="insights-gathering" containerID="cri-o://fb31b16220f88daec24acd71d67fb60a326a3b18f89e3de1b3d895fd97bbd3fb" gracePeriod=30 Oct 30 19:04:56.332815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.332785 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:56.397819 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.397740 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-trusted-ca-bundle\") pod \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " Oct 30 19:04:56.397819 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.397797 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-service-ca-bundle\") pod \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " Oct 30 19:04:56.397819 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.397821 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a7defa5-7e66-4e17-8dc8-e74137e4f054-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1a7defa5-7e66-4e17-8dc8-e74137e4f054-archives-path\") pod \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " Oct 30 19:04:56.398090 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.397871 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7defa5-7e66-4e17-8dc8-e74137e4f054-kube-api-access-gmxmw\" (UniqueName: \"kubernetes.io/projected/1a7defa5-7e66-4e17-8dc8-e74137e4f054-kube-api-access-gmxmw\") pod \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\" (UID: \"1a7defa5-7e66-4e17-8dc8-e74137e4f054\") " Oct 30 19:04:56.398223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.398150 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-service-ca-bundle" pod "1a7defa5-7e66-4e17-8dc8-e74137e4f054" (UID: "1a7defa5-7e66-4e17-8dc8-e74137e4f054"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:56.398287 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.398251 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-trusted-ca-bundle" pod "1a7defa5-7e66-4e17-8dc8-e74137e4f054" (UID: "1a7defa5-7e66-4e17-8dc8-e74137e4f054"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:56.398344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.398297 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7defa5-7e66-4e17-8dc8-e74137e4f054-archives-path" pod "1a7defa5-7e66-4e17-8dc8-e74137e4f054" (UID: "1a7defa5-7e66-4e17-8dc8-e74137e4f054"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:56.400357 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.400332 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7defa5-7e66-4e17-8dc8-e74137e4f054-kube-api-access-gmxmw" pod "1a7defa5-7e66-4e17-8dc8-e74137e4f054" (UID: "1a7defa5-7e66-4e17-8dc8-e74137e4f054"). InnerVolumeSpecName "kube-api-access-gmxmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:56.402750 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.402725 5047 generic.go:356] "Generic (PLEG): container finished" podID="1a7defa5-7e66-4e17-8dc8-e74137e4f054" containerID="fb31b16220f88daec24acd71d67fb60a326a3b18f89e3de1b3d895fd97bbd3fb" exitCode=2 Oct 30 19:04:56.402846 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.402766 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" Oct 30 19:04:56.402846 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.402803 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" event={"ID":"1a7defa5-7e66-4e17-8dc8-e74137e4f054","Type":"ContainerDied","Data":"fb31b16220f88daec24acd71d67fb60a326a3b18f89e3de1b3d895fd97bbd3fb"} Oct 30 19:04:56.402955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.402854 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hkjk5-l9rq7" event={"ID":"1a7defa5-7e66-4e17-8dc8-e74137e4f054","Type":"ContainerDied","Data":"235840f48c7642fabfb8577e4dc8aae46f92e2b241626f9eda4b30ae41a26152"} Oct 30 19:04:56.402955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.402875 5047 scope.go:122] "RemoveContainer" containerID="fb31b16220f88daec24acd71d67fb60a326a3b18f89e3de1b3d895fd97bbd3fb" Oct 30 19:04:56.411112 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.411098 5047 scope.go:122] "RemoveContainer" containerID="fb31b16220f88daec24acd71d67fb60a326a3b18f89e3de1b3d895fd97bbd3fb" Oct 30 19:04:56.411342 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:04:56.411327 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb31b16220f88daec24acd71d67fb60a326a3b18f89e3de1b3d895fd97bbd3fb\": container with ID starting with fb31b16220f88daec24acd71d67fb60a326a3b18f89e3de1b3d895fd97bbd3fb not found: ID does not exist" containerID="fb31b16220f88daec24acd71d67fb60a326a3b18f89e3de1b3d895fd97bbd3fb" Oct 30 19:04:56.411384 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.411350 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb31b16220f88daec24acd71d67fb60a326a3b18f89e3de1b3d895fd97bbd3fb"} err="failed to get container status \"fb31b16220f88daec24acd71d67fb60a326a3b18f89e3de1b3d895fd97bbd3fb\": rpc error: code = NotFound desc = could not find container \"fb31b16220f88daec24acd71d67fb60a326a3b18f89e3de1b3d895fd97bbd3fb\": container with ID starting with fb31b16220f88daec24acd71d67fb60a326a3b18f89e3de1b3d895fd97bbd3fb not found: ID does not exist" Oct 30 19:04:56.445054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.445033 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hkjk5-l9rq7"] Oct 30 19:04:56.448522 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.448503 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-hkjk5-l9rq7"] Oct 30 19:04:56.499306 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.499281 5047 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-trusted-ca-bundle\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:04:56.499306 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.499303 5047 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a7defa5-7e66-4e17-8dc8-e74137e4f054-service-ca-bundle\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:04:56.499450 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.499316 5047 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1a7defa5-7e66-4e17-8dc8-e74137e4f054-archives-path\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:04:56.499450 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:56.499325 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmxmw\" (UniqueName: \"kubernetes.io/projected/1a7defa5-7e66-4e17-8dc8-e74137e4f054-kube-api-access-gmxmw\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:04:57.046282 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:04:57.046249 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a7defa5-7e66-4e17-8dc8-e74137e4f054" path="/var/lib/kubelet/pods/1a7defa5-7e66-4e17-8dc8-e74137e4f054/volumes" Oct 30 19:05:09.599098 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:05:09.599064 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:05:11.296144 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.296075 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" podUID="3ac3ab69-eead-4eca-af11-53fec376a6ac" containerName="registry" containerID="cri-o://3786bb68ae3fc3d14d041fec47a72057f86df44d346808845bedb865e8853283" gracePeriod=30 Oct 30 19:05:11.470480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.470450 5047 generic.go:356] "Generic (PLEG): container finished" podID="3ac3ab69-eead-4eca-af11-53fec376a6ac" containerID="3786bb68ae3fc3d14d041fec47a72057f86df44d346808845bedb865e8853283" exitCode=0 Oct 30 19:05:11.470635 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.470507 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" event={"ID":"3ac3ab69-eead-4eca-af11-53fec376a6ac","Type":"ContainerDied","Data":"3786bb68ae3fc3d14d041fec47a72057f86df44d346808845bedb865e8853283"} Oct 30 19:05:11.529578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.529550 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:05:11.622386 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.622348 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-trusted-ca\") pod \"3ac3ab69-eead-4eca-af11-53fec376a6ac\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " Oct 30 19:05:11.622568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.622405 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-image-registry-private-configuration\") pod \"3ac3ab69-eead-4eca-af11-53fec376a6ac\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " Oct 30 19:05:11.622568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.622462 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-certificates\") pod \"3ac3ab69-eead-4eca-af11-53fec376a6ac\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " Oct 30 19:05:11.622568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.622494 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac3ab69-eead-4eca-af11-53fec376a6ac-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ac3ab69-eead-4eca-af11-53fec376a6ac-ca-trust-extracted\") pod \"3ac3ab69-eead-4eca-af11-53fec376a6ac\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " Oct 30 19:05:11.622568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.622518 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-bound-sa-token\") pod \"3ac3ab69-eead-4eca-af11-53fec376a6ac\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " Oct 30 19:05:11.622568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.622540 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls\") pod \"3ac3ab69-eead-4eca-af11-53fec376a6ac\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " Oct 30 19:05:11.622834 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.622592 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-kube-api-access-zn7mf\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-kube-api-access-zn7mf\") pod \"3ac3ab69-eead-4eca-af11-53fec376a6ac\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " Oct 30 19:05:11.622834 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.622640 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-installation-pull-secrets\") pod \"3ac3ab69-eead-4eca-af11-53fec376a6ac\" (UID: \"3ac3ab69-eead-4eca-af11-53fec376a6ac\") " Oct 30 19:05:11.622933 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.622819 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-trusted-ca" pod "3ac3ab69-eead-4eca-af11-53fec376a6ac" (UID: "3ac3ab69-eead-4eca-af11-53fec376a6ac"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:11.622933 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.622909 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-certificates" pod "3ac3ab69-eead-4eca-af11-53fec376a6ac" (UID: "3ac3ab69-eead-4eca-af11-53fec376a6ac"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:11.625372 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.625326 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-bound-sa-token" pod "3ac3ab69-eead-4eca-af11-53fec376a6ac" (UID: "3ac3ab69-eead-4eca-af11-53fec376a6ac"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:11.625492 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.625371 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-kube-api-access-zn7mf" pod "3ac3ab69-eead-4eca-af11-53fec376a6ac" (UID: "3ac3ab69-eead-4eca-af11-53fec376a6ac"). InnerVolumeSpecName "kube-api-access-zn7mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:11.625492 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.625371 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-image-registry-private-configuration" pod "3ac3ab69-eead-4eca-af11-53fec376a6ac" (UID: "3ac3ab69-eead-4eca-af11-53fec376a6ac"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:05:11.625608 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.625548 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-installation-pull-secrets" pod "3ac3ab69-eead-4eca-af11-53fec376a6ac" (UID: "3ac3ab69-eead-4eca-af11-53fec376a6ac"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:05:11.625608 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.625544 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls" pod "3ac3ab69-eead-4eca-af11-53fec376a6ac" (UID: "3ac3ab69-eead-4eca-af11-53fec376a6ac"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:11.629839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.629776 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac3ab69-eead-4eca-af11-53fec376a6ac-ca-trust-extracted" pod "3ac3ab69-eead-4eca-af11-53fec376a6ac" (UID: "3ac3ab69-eead-4eca-af11-53fec376a6ac"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:11.723940 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.723901 5047 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-bound-sa-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:05:11.723940 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.723934 5047 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:05:11.723940 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.723943 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn7mf\" (UniqueName: \"kubernetes.io/projected/3ac3ab69-eead-4eca-af11-53fec376a6ac-kube-api-access-zn7mf\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:05:11.724197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.723953 5047 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-installation-pull-secrets\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:05:11.724197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.723962 5047 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-trusted-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:05:11.724197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.723972 5047 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ac3ab69-eead-4eca-af11-53fec376a6ac-image-registry-private-configuration\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:05:11.724197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.723981 5047 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ac3ab69-eead-4eca-af11-53fec376a6ac-registry-certificates\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:05:11.724197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:11.723990 5047 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ac3ab69-eead-4eca-af11-53fec376a6ac-ca-trust-extracted\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:05:12.476057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:12.476022 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" event={"ID":"3ac3ab69-eead-4eca-af11-53fec376a6ac","Type":"ContainerDied","Data":"7a7329e3cf0cb878f0f65ca567d9e2189bd568474405f7975fca6e9deff8b993"} Oct 30 19:05:12.476463 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:12.476078 5047 scope.go:122] "RemoveContainer" containerID="3786bb68ae3fc3d14d041fec47a72057f86df44d346808845bedb865e8853283" Oct 30 19:05:12.476463 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:12.476036 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fd9cf966f-mn64l" Oct 30 19:05:12.529220 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:12.529191 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5fd9cf966f-mn64l"] Oct 30 19:05:12.539796 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:12.539770 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5fd9cf966f-mn64l"] Oct 30 19:05:13.045890 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:13.045861 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac3ab69-eead-4eca-af11-53fec376a6ac" path="/var/lib/kubelet/pods/3ac3ab69-eead-4eca-af11-53fec376a6ac/volumes" Oct 30 19:05:36.572672 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:36.572581 5047 generic.go:356] "Generic (PLEG): container finished" podID="11d31a53-ae1b-4d46-aa8c-6c00e89ed539" containerID="fb46dd0740e798d0d2b1e9af3a066bcaac9f3ea18a3e27ccb4194c38d60deb1c" exitCode=0 Oct 30 19:05:36.572672 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:36.572650 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wd8lm" event={"ID":"11d31a53-ae1b-4d46-aa8c-6c00e89ed539","Type":"ContainerDied","Data":"fb46dd0740e798d0d2b1e9af3a066bcaac9f3ea18a3e27ccb4194c38d60deb1c"} Oct 30 19:05:36.573292 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:36.572708 5047 scope.go:122] "RemoveContainer" containerID="81c26b6e909fc8f6eedbfe63cb64a1fe1e7a5f03c1e96645094aaf67c2c245d0" Oct 30 19:05:36.573292 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:36.573112 5047 scope.go:122] "RemoveContainer" containerID="fb46dd0740e798d0d2b1e9af3a066bcaac9f3ea18a3e27ccb4194c38d60deb1c" Oct 30 19:05:36.573395 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:05:36.573375 5047 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-wd8lm_openshift-insights(11d31a53-ae1b-4d46-aa8c-6c00e89ed539)\"" pod="openshift-insights/insights-operator-7c4986d449-wd8lm" podUID="11d31a53-ae1b-4d46-aa8c-6c00e89ed539" Oct 30 19:05:39.599770 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:05:39.599734 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:05:49.624864 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:49.624829 5047 generic.go:356] "Generic (PLEG): container finished" podID="9c46dbf4-49e2-4435-8362-255dcd09ebbf" containerID="e22993b6753f3d7ad3c27b5439c08e60c221206b1b8234589191bfa9efc840fd" exitCode=0 Oct 30 19:05:49.625222 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:49.624881 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-m5mwl-99mnf" event={"ID":"9c46dbf4-49e2-4435-8362-255dcd09ebbf","Type":"ContainerDied","Data":"e22993b6753f3d7ad3c27b5439c08e60c221206b1b8234589191bfa9efc840fd"} Oct 30 19:05:50.041374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:50.041344 5047 scope.go:122] "RemoveContainer" containerID="fb46dd0740e798d0d2b1e9af3a066bcaac9f3ea18a3e27ccb4194c38d60deb1c" Oct 30 19:05:50.631192 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:50.631154 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wd8lm" event={"ID":"11d31a53-ae1b-4d46-aa8c-6c00e89ed539","Type":"ContainerStarted","Data":"b067aab5dc12813448261875c3599f2af525cbd89a1cea1e906159360bdc93f0"} Oct 30 19:05:50.662471 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:50.662439 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-m5mwl-99mnf"] Oct 30 19:05:50.776151 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:50.776127 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-m5mwl-99mnf" Oct 30 19:05:50.833112 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:50.833085 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c46dbf4-49e2-4435-8362-255dcd09ebbf-kube-api-access-cxvxm\" (UniqueName: \"kubernetes.io/projected/9c46dbf4-49e2-4435-8362-255dcd09ebbf-kube-api-access-cxvxm\") pod \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\" (UID: \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\") " Oct 30 19:05:50.833415 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:50.833140 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c46dbf4-49e2-4435-8362-255dcd09ebbf-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46dbf4-49e2-4435-8362-255dcd09ebbf-service-ca-bundle\") pod \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\" (UID: \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\") " Oct 30 19:05:50.833415 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:50.833162 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c46dbf4-49e2-4435-8362-255dcd09ebbf-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9c46dbf4-49e2-4435-8362-255dcd09ebbf-archives-path\") pod \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\" (UID: \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\") " Oct 30 19:05:50.833415 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:50.833220 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c46dbf4-49e2-4435-8362-255dcd09ebbf-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46dbf4-49e2-4435-8362-255dcd09ebbf-trusted-ca-bundle\") pod \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\" (UID: \"9c46dbf4-49e2-4435-8362-255dcd09ebbf\") " Oct 30 19:05:50.833598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:50.833515 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c46dbf4-49e2-4435-8362-255dcd09ebbf-service-ca-bundle" pod "9c46dbf4-49e2-4435-8362-255dcd09ebbf" (UID: "9c46dbf4-49e2-4435-8362-255dcd09ebbf"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:50.833598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:50.833525 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c46dbf4-49e2-4435-8362-255dcd09ebbf-archives-path" pod "9c46dbf4-49e2-4435-8362-255dcd09ebbf" (UID: "9c46dbf4-49e2-4435-8362-255dcd09ebbf"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:50.833598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:50.833588 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c46dbf4-49e2-4435-8362-255dcd09ebbf-trusted-ca-bundle" pod "9c46dbf4-49e2-4435-8362-255dcd09ebbf" (UID: "9c46dbf4-49e2-4435-8362-255dcd09ebbf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:50.835080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:50.835056 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c46dbf4-49e2-4435-8362-255dcd09ebbf-kube-api-access-cxvxm" pod "9c46dbf4-49e2-4435-8362-255dcd09ebbf" (UID: "9c46dbf4-49e2-4435-8362-255dcd09ebbf"). InnerVolumeSpecName "kube-api-access-cxvxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:50.934568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:50.934500 5047 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46dbf4-49e2-4435-8362-255dcd09ebbf-service-ca-bundle\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:05:50.934568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:50.934535 5047 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9c46dbf4-49e2-4435-8362-255dcd09ebbf-archives-path\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:05:50.934568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:50.934545 5047 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c46dbf4-49e2-4435-8362-255dcd09ebbf-trusted-ca-bundle\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:05:50.934568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:50.934554 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxvxm\" (UniqueName: \"kubernetes.io/projected/9c46dbf4-49e2-4435-8362-255dcd09ebbf-kube-api-access-cxvxm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:05:51.636047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:51.636001 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-m5mwl-99mnf" Oct 30 19:05:51.636047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:51.636032 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-m5mwl-99mnf" event={"ID":"9c46dbf4-49e2-4435-8362-255dcd09ebbf","Type":"ContainerDied","Data":"bff0398fb79c0c472cfe3282517100f2415acf9733bda4923750aaa5c1dd6172"} Oct 30 19:05:51.636479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:51.636074 5047 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bff0398fb79c0c472cfe3282517100f2415acf9733bda4923750aaa5c1dd6172" Oct 30 19:05:51.699659 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:51.699621 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-m5mwl-99mnf"] Oct 30 19:05:51.703457 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:51.703428 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-m5mwl-99mnf"] Oct 30 19:05:53.045981 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:05:53.045948 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c46dbf4-49e2-4435-8362-255dcd09ebbf" path="/var/lib/kubelet/pods/9c46dbf4-49e2-4435-8362-255dcd09ebbf/volumes" Oct 30 19:06:09.598929 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:06:09.598884 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:06:39.598792 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:06:39.598715 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:07:09.599062 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:07:09.599025 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:07:39.598503 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:07:39.598463 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:08:09.599500 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:08:09.599460 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:08:39.599815 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:08:39.599773 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:09:07.067618 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:09:07.067592 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:09:07.068104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:09:07.067637 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:09:09.599393 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:09:09.599351 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:09:39.599723 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:09:39.599639 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:10:09.600087 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:10:09.600045 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:10:39.599223 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:10:39.599187 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:10:55.293297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.293260 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-q2thr"] Oct 30 19:10:55.293794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.293594 5047 cpu_manager.go:414] "containerMap: removing container" podUID="3ac3ab69-eead-4eca-af11-53fec376a6ac" containerName="registry" Oct 30 19:10:55.293794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.293607 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac3ab69-eead-4eca-af11-53fec376a6ac" containerName="registry" Oct 30 19:10:55.293794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.293628 5047 cpu_manager.go:414] "containerMap: removing container" podUID="9c46dbf4-49e2-4435-8362-255dcd09ebbf" containerName="insights-gathering" Oct 30 19:10:55.293794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.293633 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c46dbf4-49e2-4435-8362-255dcd09ebbf" containerName="insights-gathering" Oct 30 19:10:55.293794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.293642 5047 cpu_manager.go:414] "containerMap: removing container" podUID="6af08cb7-fa60-491b-88fd-07c1ace45779" containerName="registry" Oct 30 19:10:55.293794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.293647 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af08cb7-fa60-491b-88fd-07c1ace45779" containerName="registry" Oct 30 19:10:55.293794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.293653 5047 cpu_manager.go:414] "containerMap: removing container" podUID="1a7defa5-7e66-4e17-8dc8-e74137e4f054" containerName="insights-gathering" Oct 30 19:10:55.293794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.293658 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7defa5-7e66-4e17-8dc8-e74137e4f054" containerName="insights-gathering" Oct 30 19:10:55.293794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.293718 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af08cb7-fa60-491b-88fd-07c1ace45779" containerName="registry" Oct 30 19:10:55.293794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.293727 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7defa5-7e66-4e17-8dc8-e74137e4f054" containerName="insights-gathering" Oct 30 19:10:55.293794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.293736 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac3ab69-eead-4eca-af11-53fec376a6ac" containerName="registry" Oct 30 19:10:55.293794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.293744 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c46dbf4-49e2-4435-8362-255dcd09ebbf" containerName="insights-gathering" Oct 30 19:10:55.296122 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.296100 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-q2thr" Oct 30 19:10:55.322309 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.322279 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-ww48c\"" Oct 30 19:10:55.322309 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.322292 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:10:55.322309 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.322304 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 19:10:55.323314 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.323291 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-q2thr"] Oct 30 19:10:55.335264 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.335241 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdlsn\" (UniqueName: \"kubernetes.io/projected/21e9ae46-ccd9-410e-ad31-f13872b3ab36-kube-api-access-jdlsn\") pod \"openshift-lws-operator-8465994787-q2thr\" (UID: \"21e9ae46-ccd9-410e-ad31-f13872b3ab36\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-q2thr" Oct 30 19:10:55.335380 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.335279 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21e9ae46-ccd9-410e-ad31-f13872b3ab36-tmp\") pod \"openshift-lws-operator-8465994787-q2thr\" (UID: \"21e9ae46-ccd9-410e-ad31-f13872b3ab36\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-q2thr" Oct 30 19:10:55.436644 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.436610 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdlsn\" (UniqueName: \"kubernetes.io/projected/21e9ae46-ccd9-410e-ad31-f13872b3ab36-kube-api-access-jdlsn\") pod \"openshift-lws-operator-8465994787-q2thr\" (UID: \"21e9ae46-ccd9-410e-ad31-f13872b3ab36\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-q2thr" Oct 30 19:10:55.436802 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.436654 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21e9ae46-ccd9-410e-ad31-f13872b3ab36-tmp\") pod \"openshift-lws-operator-8465994787-q2thr\" (UID: \"21e9ae46-ccd9-410e-ad31-f13872b3ab36\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-q2thr" Oct 30 19:10:55.436976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.436950 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21e9ae46-ccd9-410e-ad31-f13872b3ab36-tmp\") pod \"openshift-lws-operator-8465994787-q2thr\" (UID: \"21e9ae46-ccd9-410e-ad31-f13872b3ab36\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-q2thr" Oct 30 19:10:55.454791 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.454765 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdlsn\" (UniqueName: \"kubernetes.io/projected/21e9ae46-ccd9-410e-ad31-f13872b3ab36-kube-api-access-jdlsn\") pod \"openshift-lws-operator-8465994787-q2thr\" (UID: \"21e9ae46-ccd9-410e-ad31-f13872b3ab36\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-q2thr" Oct 30 19:10:55.618560 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.618525 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-q2thr" Oct 30 19:10:55.904983 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.904843 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-q2thr"] Oct 30 19:10:55.907715 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:10:55.907686 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21e9ae46_ccd9_410e_ad31_f13872b3ab36.slice/crio-6edbc0db8766f554b18fa3ccbdcefcf58ddc34fe8712ee85f89a1da24484b369 WatchSource:0}: Error finding container 6edbc0db8766f554b18fa3ccbdcefcf58ddc34fe8712ee85f89a1da24484b369: Status 404 returned error can't find the container with id 6edbc0db8766f554b18fa3ccbdcefcf58ddc34fe8712ee85f89a1da24484b369 Oct 30 19:10:55.909308 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:55.909289 5047 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:10:56.797467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:56.797429 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-q2thr" event={"ID":"21e9ae46-ccd9-410e-ad31-f13872b3ab36","Type":"ContainerStarted","Data":"6edbc0db8766f554b18fa3ccbdcefcf58ddc34fe8712ee85f89a1da24484b369"} Oct 30 19:10:59.812166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:59.812124 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-q2thr" event={"ID":"21e9ae46-ccd9-410e-ad31-f13872b3ab36","Type":"ContainerStarted","Data":"5895aac430a4a82ddafe0f87d1066c3b97bee77055d65f909d8eb6795b8c9c5a"} Oct 30 19:10:59.848454 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:10:59.848396 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-q2thr" podStartSLOduration=1.882083727 podStartE2EDuration="4.848381427s" totalImagesPullingTime="2.9662977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:59.847382615 +0000 UTC m=+412.921837974" watchObservedRunningTime="2026-10-30 19:10:59.848381427 +0000 UTC m=+412.922836789" Oct 30 19:11:07.059988 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:07.059957 5047 scope.go:122] "RemoveContainer" containerID="e22993b6753f3d7ad3c27b5439c08e60c221206b1b8234589191bfa9efc840fd" Oct 30 19:11:09.599448 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:11:09.599404 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:11:14.310632 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.310588 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865"] Oct 30 19:11:14.316284 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.316257 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.348633 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.348594 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865"] Oct 30 19:11:14.365277 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.365247 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 19:11:14.365441 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.365396 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 19:11:14.365441 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.365421 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 19:11:14.365441 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.365428 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 19:11:14.365544 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.365479 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 19:11:14.365576 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.365547 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2lq27\"" Oct 30 19:11:14.385667 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.385634 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dfe5c738-462f-461b-b1e9-1f449896ed4e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.385848 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.385682 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfe5c738-462f-461b-b1e9-1f449896ed4e-tmp\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.385848 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.385791 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dfe5c738-462f-461b-b1e9-1f449896ed4e-metrics-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.385848 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.385834 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dfe5c738-462f-461b-b1e9-1f449896ed4e-manifests\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.385961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.385903 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps6cq\" (UniqueName: \"kubernetes.io/projected/dfe5c738-462f-461b-b1e9-1f449896ed4e-kube-api-access-ps6cq\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.385961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.385930 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfe5c738-462f-461b-b1e9-1f449896ed4e-webhook-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.487196 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.487159 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfe5c738-462f-461b-b1e9-1f449896ed4e-tmp\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.487196 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.487199 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dfe5c738-462f-461b-b1e9-1f449896ed4e-metrics-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.487411 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.487220 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dfe5c738-462f-461b-b1e9-1f449896ed4e-manifests\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.487411 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.487253 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps6cq\" (UniqueName: \"kubernetes.io/projected/dfe5c738-462f-461b-b1e9-1f449896ed4e-kube-api-access-ps6cq\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.487411 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.487279 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfe5c738-462f-461b-b1e9-1f449896ed4e-webhook-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.487411 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.487306 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dfe5c738-462f-461b-b1e9-1f449896ed4e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.487720 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.487698 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dfe5c738-462f-461b-b1e9-1f449896ed4e-manifests\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.489672 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.489645 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfe5c738-462f-461b-b1e9-1f449896ed4e-tmp\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.489926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.489906 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dfe5c738-462f-461b-b1e9-1f449896ed4e-webhook-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.489976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.489933 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dfe5c738-462f-461b-b1e9-1f449896ed4e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.489976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.489947 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dfe5c738-462f-461b-b1e9-1f449896ed4e-metrics-cert\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.506285 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.506262 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps6cq\" (UniqueName: \"kubernetes.io/projected/dfe5c738-462f-461b-b1e9-1f449896ed4e-kube-api-access-ps6cq\") pod \"opendatahub-operator-controller-manager-8589ff54bc-ch865\" (UID: \"dfe5c738-462f-461b-b1e9-1f449896ed4e\") " pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.630368 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.630329 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:14.983671 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:14.983648 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865"] Oct 30 19:11:14.985348 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:11:14.985317 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe5c738_462f_461b_b1e9_1f449896ed4e.slice/crio-17433a8c2cb2e935bec29ddd8df9398240cbe24fc7421ab00816f860a7ec562c WatchSource:0}: Error finding container 17433a8c2cb2e935bec29ddd8df9398240cbe24fc7421ab00816f860a7ec562c: Status 404 returned error can't find the container with id 17433a8c2cb2e935bec29ddd8df9398240cbe24fc7421ab00816f860a7ec562c Oct 30 19:11:15.877333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:15.877247 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" event={"ID":"dfe5c738-462f-461b-b1e9-1f449896ed4e","Type":"ContainerStarted","Data":"17433a8c2cb2e935bec29ddd8df9398240cbe24fc7421ab00816f860a7ec562c"} Oct 30 19:11:17.887701 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:17.887657 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" event={"ID":"dfe5c738-462f-461b-b1e9-1f449896ed4e","Type":"ContainerStarted","Data":"087e9c902a6cc474738a8f9ea6149f9c1daebd8bc5754a0e4ce3c929987c8be9"} Oct 30 19:11:18.893478 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:18.893441 5047 generic.go:356] "Generic (PLEG): container finished" podID="dfe5c738-462f-461b-b1e9-1f449896ed4e" containerID="087e9c902a6cc474738a8f9ea6149f9c1daebd8bc5754a0e4ce3c929987c8be9" exitCode=0 Oct 30 19:11:18.893983 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:18.893529 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" event={"ID":"dfe5c738-462f-461b-b1e9-1f449896ed4e","Type":"ContainerDied","Data":"087e9c902a6cc474738a8f9ea6149f9c1daebd8bc5754a0e4ce3c929987c8be9"} Oct 30 19:11:19.900066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:19.900030 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" event={"ID":"dfe5c738-462f-461b-b1e9-1f449896ed4e","Type":"ContainerStarted","Data":"78804c11bde292d110018c239f469d5cad8eb782bebd4d077d2a6a0ac5ec1133"} Oct 30 19:11:19.900546 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:19.900163 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:19.973870 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:19.973815 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" podStartSLOduration=2.449273818 podStartE2EDuration="5.973800027s" totalImagesPullingTime="3.116764654s" totalInitContainerRuntime="407.761555ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:19.963276293 +0000 UTC m=+433.037731653" watchObservedRunningTime="2026-10-30 19:11:19.973800027 +0000 UTC m=+433.048255383" Oct 30 19:11:30.906499 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:30.906469 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8589ff54bc-ch865" Oct 30 19:11:37.281849 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:37.281816 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-564b7"] Oct 30 19:11:37.290961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:37.290941 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-564b7" Oct 30 19:11:37.303357 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:37.303338 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\"" Oct 30 19:11:37.313703 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:37.313678 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-dd4mg\"" Oct 30 19:11:37.314682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:37.314661 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-564b7"] Oct 30 19:11:37.487945 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:37.487904 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq7rm\" (UniqueName: \"kubernetes.io/projected/0f369006-b3bf-412c-9c5b-90e2f945bc76-kube-api-access-gq7rm\") pod \"odh-observability-584f99c788-564b7\" (UID: \"0f369006-b3bf-412c-9c5b-90e2f945bc76\") " pod="opendatahub/odh-observability-584f99c788-564b7" Oct 30 19:11:37.488140 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:37.487978 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/0f369006-b3bf-412c-9c5b-90e2f945bc76-webhook-certs\") pod \"odh-observability-584f99c788-564b7\" (UID: \"0f369006-b3bf-412c-9c5b-90e2f945bc76\") " pod="opendatahub/odh-observability-584f99c788-564b7" Oct 30 19:11:37.589069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:37.589039 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq7rm\" (UniqueName: \"kubernetes.io/projected/0f369006-b3bf-412c-9c5b-90e2f945bc76-kube-api-access-gq7rm\") pod \"odh-observability-584f99c788-564b7\" (UID: \"0f369006-b3bf-412c-9c5b-90e2f945bc76\") " pod="opendatahub/odh-observability-584f99c788-564b7" Oct 30 19:11:37.589230 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:37.589104 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/0f369006-b3bf-412c-9c5b-90e2f945bc76-webhook-certs\") pod \"odh-observability-584f99c788-564b7\" (UID: \"0f369006-b3bf-412c-9c5b-90e2f945bc76\") " pod="opendatahub/odh-observability-584f99c788-564b7" Oct 30 19:11:37.591495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:37.591468 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/0f369006-b3bf-412c-9c5b-90e2f945bc76-webhook-certs\") pod \"odh-observability-584f99c788-564b7\" (UID: \"0f369006-b3bf-412c-9c5b-90e2f945bc76\") " pod="opendatahub/odh-observability-584f99c788-564b7" Oct 30 19:11:37.606982 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:37.606956 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq7rm\" (UniqueName: \"kubernetes.io/projected/0f369006-b3bf-412c-9c5b-90e2f945bc76-kube-api-access-gq7rm\") pod \"odh-observability-584f99c788-564b7\" (UID: \"0f369006-b3bf-412c-9c5b-90e2f945bc76\") " pod="opendatahub/odh-observability-584f99c788-564b7" Oct 30 19:11:37.902069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:37.901942 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-564b7" Oct 30 19:11:38.189422 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:38.189398 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-564b7"] Oct 30 19:11:38.192032 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:11:38.191988 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f369006_b3bf_412c_9c5b_90e2f945bc76.slice/crio-c0cda96c2ff2cc3bdbce82985fb8eaf0eeed202ee6d965c315acd9b768f261cd WatchSource:0}: Error finding container c0cda96c2ff2cc3bdbce82985fb8eaf0eeed202ee6d965c315acd9b768f261cd: Status 404 returned error can't find the container with id c0cda96c2ff2cc3bdbce82985fb8eaf0eeed202ee6d965c315acd9b768f261cd Oct 30 19:11:38.973956 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:38.973920 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-564b7" event={"ID":"0f369006-b3bf-412c-9c5b-90e2f945bc76","Type":"ContainerStarted","Data":"c0cda96c2ff2cc3bdbce82985fb8eaf0eeed202ee6d965c315acd9b768f261cd"} Oct 30 19:11:39.604873 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:11:39.604838 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:11:40.984060 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:40.984021 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-564b7" event={"ID":"0f369006-b3bf-412c-9c5b-90e2f945bc76","Type":"ContainerStarted","Data":"a52a961b1d10f90f2763b9521490d3877d56a9465bab4306d4a22dd846e6d39e"} Oct 30 19:11:40.984462 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:40.984125 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-564b7" Oct 30 19:11:41.019231 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:41.019171 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-564b7" podStartSLOduration=1.413115783 podStartE2EDuration="4.019154965s" totalImagesPullingTime="2.606039182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:41.018206329 +0000 UTC m=+454.092661689" watchObservedRunningTime="2026-10-30 19:11:41.019154965 +0000 UTC m=+454.093610323" Oct 30 19:11:45.205907 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:45.205872 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk"] Oct 30 19:11:45.208401 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:45.208384 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk" Oct 30 19:11:45.220748 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:45.220720 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 19:11:45.220869 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:45.220786 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 19:11:45.230966 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:45.230941 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wmzc4\"" Oct 30 19:11:45.232078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:45.232054 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk"] Oct 30 19:11:45.244878 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:45.244855 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2298b\" (UniqueName: \"kubernetes.io/projected/eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d-kube-api-access-2298b\") pod \"kube-auth-proxy-6c7d47d6c9-jvqgk\" (UID: \"eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk" Oct 30 19:11:45.244977 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:45.244906 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d-tmp\") pod \"kube-auth-proxy-6c7d47d6c9-jvqgk\" (UID: \"eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk" Oct 30 19:11:45.245080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:45.245056 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d-tls-certs\") pod \"kube-auth-proxy-6c7d47d6c9-jvqgk\" (UID: \"eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk" Oct 30 19:11:45.345386 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:45.345350 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d-tmp\") pod \"kube-auth-proxy-6c7d47d6c9-jvqgk\" (UID: \"eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk" Oct 30 19:11:45.345537 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:45.345395 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d-tls-certs\") pod \"kube-auth-proxy-6c7d47d6c9-jvqgk\" (UID: \"eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk" Oct 30 19:11:45.345537 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:45.345452 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2298b\" (UniqueName: \"kubernetes.io/projected/eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d-kube-api-access-2298b\") pod \"kube-auth-proxy-6c7d47d6c9-jvqgk\" (UID: \"eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk" Oct 30 19:11:45.347550 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:45.347529 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d-tmp\") pod \"kube-auth-proxy-6c7d47d6c9-jvqgk\" (UID: \"eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk" Oct 30 19:11:45.347868 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:45.347845 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d-tls-certs\") pod \"kube-auth-proxy-6c7d47d6c9-jvqgk\" (UID: \"eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk" Oct 30 19:11:45.363372 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:45.363346 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2298b\" (UniqueName: \"kubernetes.io/projected/eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d-kube-api-access-2298b\") pod \"kube-auth-proxy-6c7d47d6c9-jvqgk\" (UID: \"eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d\") " pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk" Oct 30 19:11:45.521554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:45.521460 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk" Oct 30 19:11:45.809669 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:45.809641 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk"] Oct 30 19:11:45.812303 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:11:45.812274 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee3ec1c_3cb4_46c8_9e8b_25ca31925e6d.slice/crio-9d2d9f1c5698bf5f7deea5fc165b4dcf58a3efd6edd5d99c67fa0ec2295c8d37 WatchSource:0}: Error finding container 9d2d9f1c5698bf5f7deea5fc165b4dcf58a3efd6edd5d99c67fa0ec2295c8d37: Status 404 returned error can't find the container with id 9d2d9f1c5698bf5f7deea5fc165b4dcf58a3efd6edd5d99c67fa0ec2295c8d37 Oct 30 19:11:46.004813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.004776 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk" event={"ID":"eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d","Type":"ContainerStarted","Data":"9d2d9f1c5698bf5f7deea5fc165b4dcf58a3efd6edd5d99c67fa0ec2295c8d37"} Oct 30 19:11:46.569180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.569144 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g"] Oct 30 19:11:46.571737 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.571707 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.585942 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.585906 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 19:11:46.585942 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.585936 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 19:11:46.586163 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.585949 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 19:11:46.586219 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.586162 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4lz7b\"" Oct 30 19:11:46.597999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.597934 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g"] Oct 30 19:11:46.637827 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.637794 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 19:11:46.655386 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.655344 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjvdp\" (UniqueName: \"kubernetes.io/projected/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-kube-api-access-bjvdp\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.655386 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.655390 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.655613 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.655541 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.655613 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.655600 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.655726 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.655663 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.655893 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.655763 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.655893 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.655817 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.756543 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.756494 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.756734 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.756648 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.756734 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.756681 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.756734 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.756714 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.756904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.756757 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.756904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.756788 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjvdp\" (UniqueName: \"kubernetes.io/projected/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-kube-api-access-bjvdp\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.756904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.756820 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.757250 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.757200 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.759856 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.759825 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.759980 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.759927 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.760064 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.760038 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.761822 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.761800 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.774531 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.774502 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.775191 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.775168 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjvdp\" (UniqueName: \"kubernetes.io/projected/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-kube-api-access-bjvdp\") pod \"istiod-openshift-gateway-7876bfcb74-5rm8g\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:46.887910 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:46.887878 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:47.198070 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:47.198044 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g"] Oct 30 19:11:47.200639 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:11:47.200607 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fa5b58_d1cb_447e_a1a6_f1c958569cb9.slice/crio-0e8b4b4adc21c355f1666b83df4aeae48b472f188d64cae8a5d20878e623b9d8 WatchSource:0}: Error finding container 0e8b4b4adc21c355f1666b83df4aeae48b472f188d64cae8a5d20878e623b9d8: Status 404 returned error can't find the container with id 0e8b4b4adc21c355f1666b83df4aeae48b472f188d64cae8a5d20878e623b9d8 Oct 30 19:11:48.016966 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:48.016925 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" event={"ID":"10fa5b58-d1cb-447e-a1a6-f1c958569cb9","Type":"ContainerStarted","Data":"0e8b4b4adc21c355f1666b83df4aeae48b472f188d64cae8a5d20878e623b9d8"} Oct 30 19:11:49.023489 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:49.023445 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk" event={"ID":"eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d","Type":"ContainerStarted","Data":"393681cd365c213f3db7d29ced4dfd20373cc740dc48388ea5b4e691db8fb403"} Oct 30 19:11:49.059291 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:49.059228 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6c7d47d6c9-jvqgk" podStartSLOduration=1.6091836590000002 podStartE2EDuration="4.059212037s" totalImagesPullingTime="2.450028378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:49.05893784 +0000 UTC m=+462.133393199" watchObservedRunningTime="2026-10-30 19:11:49.059212037 +0000 UTC m=+462.133667399" Oct 30 19:11:49.808824 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:49.808791 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:11:49.808930 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:49.808854 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:11:50.029205 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:50.029164 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" event={"ID":"10fa5b58-d1cb-447e-a1a6-f1c958569cb9","Type":"ContainerStarted","Data":"35b75f08913ede59680b9233f408e8c08cfcaf01e12a565874e71becc8372e16"} Oct 30 19:11:50.076226 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:50.076120 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" podStartSLOduration=1.471214061 podStartE2EDuration="4.076103823s" totalImagesPullingTime="2.604889762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:50.075554675 +0000 UTC m=+463.150010037" watchObservedRunningTime="2026-10-30 19:11:50.076103823 +0000 UTC m=+463.150559181" Oct 30 19:11:51.034659 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:51.034627 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:11:51.036079 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:51.036044 5047 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-5rm8g container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:11:51.036201 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:51.036090 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" podUID="10fa5b58-d1cb-447e-a1a6-f1c958569cb9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:11:51.990643 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:51.990615 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-564b7" Oct 30 19:11:52.039596 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:11:52.039571 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:12:09.598975 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:12:09.598933 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:12:35.633975 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.633868 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv"] Oct 30 19:12:35.640798 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.640774 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.659083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.658971 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv"] Oct 30 19:12:35.682683 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.682651 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d62bc31-309d-4724-8bda-83f24f229a2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.682829 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.682776 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d62bc31-309d-4724-8bda-83f24f229a2e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.682886 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.682846 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n645f\" (UniqueName: \"kubernetes.io/projected/7d62bc31-309d-4724-8bda-83f24f229a2e-kube-api-access-n645f\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.682940 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.682893 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7d62bc31-309d-4724-8bda-83f24f229a2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.683049 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.683030 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7d62bc31-309d-4724-8bda-83f24f229a2e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.683110 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.683061 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7d62bc31-309d-4724-8bda-83f24f229a2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.683157 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.683135 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d62bc31-309d-4724-8bda-83f24f229a2e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.783801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.783761 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7d62bc31-309d-4724-8bda-83f24f229a2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.783801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.783800 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7d62bc31-309d-4724-8bda-83f24f229a2e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.784086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.783820 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7d62bc31-309d-4724-8bda-83f24f229a2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.784086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.783842 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d62bc31-309d-4724-8bda-83f24f229a2e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.784086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.783870 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d62bc31-309d-4724-8bda-83f24f229a2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.784086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.783933 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d62bc31-309d-4724-8bda-83f24f229a2e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.784086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.783974 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n645f\" (UniqueName: \"kubernetes.io/projected/7d62bc31-309d-4724-8bda-83f24f229a2e-kube-api-access-n645f\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.784689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.784658 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7d62bc31-309d-4724-8bda-83f24f229a2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.786327 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.786295 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d62bc31-309d-4724-8bda-83f24f229a2e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.786430 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.786367 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7d62bc31-309d-4724-8bda-83f24f229a2e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.786542 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.786520 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7d62bc31-309d-4724-8bda-83f24f229a2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.786583 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.786535 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d62bc31-309d-4724-8bda-83f24f229a2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.801220 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.801197 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d62bc31-309d-4724-8bda-83f24f229a2e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.801596 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.801574 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n645f\" (UniqueName: \"kubernetes.io/projected/7d62bc31-309d-4724-8bda-83f24f229a2e-kube-api-access-n645f\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzmjv\" (UID: \"7d62bc31-309d-4724-8bda-83f24f229a2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:35.951333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:35.951251 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:36.279494 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:36.279401 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv"] Oct 30 19:12:36.282203 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:12:36.282178 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d62bc31_309d_4724_8bda_83f24f229a2e.slice/crio-afd85df13bd6abf9f60e5be33a2c6f6fb4ae75880f2ba6efe7775a2ebb2442c9 WatchSource:0}: Error finding container afd85df13bd6abf9f60e5be33a2c6f6fb4ae75880f2ba6efe7775a2ebb2442c9: Status 404 returned error can't find the container with id afd85df13bd6abf9f60e5be33a2c6f6fb4ae75880f2ba6efe7775a2ebb2442c9 Oct 30 19:12:36.284591 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:36.284561 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:36.284681 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:36.284619 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:37.224808 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:37.224765 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" event={"ID":"7d62bc31-309d-4724-8bda-83f24f229a2e","Type":"ContainerStarted","Data":"b68976007b8843e8e680fe2cb562336c2b71820e54e80e0f1e40517446122345"} Oct 30 19:12:37.225446 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:37.224834 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" event={"ID":"7d62bc31-309d-4724-8bda-83f24f229a2e","Type":"ContainerStarted","Data":"afd85df13bd6abf9f60e5be33a2c6f6fb4ae75880f2ba6efe7775a2ebb2442c9"} Oct 30 19:12:37.225446 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:37.224963 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:37.227124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:37.227094 5047 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kzmjv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:12:37.227239 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:37.227139 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" podUID="7d62bc31-309d-4724-8bda-83f24f229a2e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:12:37.270874 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:37.270810 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" podStartSLOduration=2.270791537 podStartE2EDuration="2.270791537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:37.265238585 +0000 UTC m=+510.339693963" watchObservedRunningTime="2026-10-30 19:12:37.270791537 +0000 UTC m=+510.345246897" Oct 30 19:12:38.230246 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.230219 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzmjv" Oct 30 19:12:38.310399 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.310363 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g"] Oct 30 19:12:38.310634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.310589 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" podUID="10fa5b58-d1cb-447e-a1a6-f1c958569cb9" containerName="discovery" containerID="cri-o://35b75f08913ede59680b9233f408e8c08cfcaf01e12a565874e71becc8372e16" gracePeriod=30 Oct 30 19:12:38.553406 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.553381 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:12:38.606882 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.606850 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-token\" (UniqueName: \"kubernetes.io/projected/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-token\") pod \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " Oct 30 19:12:38.607046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.606896 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-kubeconfig\") pod \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " Oct 30 19:12:38.607046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.606921 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-cacerts\" (UniqueName: \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-cacerts\") pod \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " Oct 30 19:12:38.607046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.606952 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-csr-dns-cert\") pod \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " Oct 30 19:12:38.607046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.606990 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-local-certs\") pod \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " Oct 30 19:12:38.607265 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.607068 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-csr-ca-configmap\") pod \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " Oct 30 19:12:38.607265 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.607106 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-kube-api-access-bjvdp\" (UniqueName: \"kubernetes.io/projected/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-kube-api-access-bjvdp\") pod \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\" (UID: \"10fa5b58-d1cb-447e-a1a6-f1c958569cb9\") " Oct 30 19:12:38.607622 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.607553 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-csr-ca-configmap" pod "10fa5b58-d1cb-447e-a1a6-f1c958569cb9" (UID: "10fa5b58-d1cb-447e-a1a6-f1c958569cb9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:38.609763 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.609728 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-local-certs" pod "10fa5b58-d1cb-447e-a1a6-f1c958569cb9" (UID: "10fa5b58-d1cb-447e-a1a6-f1c958569cb9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:38.609973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.609943 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-kubeconfig" pod "10fa5b58-d1cb-447e-a1a6-f1c958569cb9" (UID: "10fa5b58-d1cb-447e-a1a6-f1c958569cb9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:12:38.609973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.609958 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-cacerts" pod "10fa5b58-d1cb-447e-a1a6-f1c958569cb9" (UID: "10fa5b58-d1cb-447e-a1a6-f1c958569cb9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:12:38.610167 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.609949 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-token" pod "10fa5b58-d1cb-447e-a1a6-f1c958569cb9" (UID: "10fa5b58-d1cb-447e-a1a6-f1c958569cb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:38.610167 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.609983 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-csr-dns-cert" pod "10fa5b58-d1cb-447e-a1a6-f1c958569cb9" (UID: "10fa5b58-d1cb-447e-a1a6-f1c958569cb9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:12:38.610167 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.610138 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-kube-api-access-bjvdp" pod "10fa5b58-d1cb-447e-a1a6-f1c958569cb9" (UID: "10fa5b58-d1cb-447e-a1a6-f1c958569cb9"). InnerVolumeSpecName "kube-api-access-bjvdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:38.707855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.707825 5047 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-local-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:12:38.707855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.707851 5047 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-csr-ca-configmap\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:12:38.707855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.707861 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjvdp\" (UniqueName: \"kubernetes.io/projected/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-kube-api-access-bjvdp\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:12:38.708114 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.707869 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:12:38.708114 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.707877 5047 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-kubeconfig\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:12:38.708114 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.707885 5047 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-cacerts\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:12:38.708114 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:38.707895 5047 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10fa5b58-d1cb-447e-a1a6-f1c958569cb9-istio-csr-dns-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:12:39.235712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:39.235681 5047 generic.go:356] "Generic (PLEG): container finished" podID="10fa5b58-d1cb-447e-a1a6-f1c958569cb9" containerID="35b75f08913ede59680b9233f408e8c08cfcaf01e12a565874e71becc8372e16" exitCode=0 Oct 30 19:12:39.236180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:39.235735 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" Oct 30 19:12:39.236180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:39.235765 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" event={"ID":"10fa5b58-d1cb-447e-a1a6-f1c958569cb9","Type":"ContainerDied","Data":"35b75f08913ede59680b9233f408e8c08cfcaf01e12a565874e71becc8372e16"} Oct 30 19:12:39.236180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:39.235813 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g" event={"ID":"10fa5b58-d1cb-447e-a1a6-f1c958569cb9","Type":"ContainerDied","Data":"0e8b4b4adc21c355f1666b83df4aeae48b472f188d64cae8a5d20878e623b9d8"} Oct 30 19:12:39.236180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:39.235834 5047 scope.go:122] "RemoveContainer" containerID="35b75f08913ede59680b9233f408e8c08cfcaf01e12a565874e71becc8372e16" Oct 30 19:12:39.245537 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:39.245509 5047 scope.go:122] "RemoveContainer" containerID="35b75f08913ede59680b9233f408e8c08cfcaf01e12a565874e71becc8372e16" Oct 30 19:12:39.245816 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:12:39.245798 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b75f08913ede59680b9233f408e8c08cfcaf01e12a565874e71becc8372e16\": container with ID starting with 35b75f08913ede59680b9233f408e8c08cfcaf01e12a565874e71becc8372e16 not found: ID does not exist" containerID="35b75f08913ede59680b9233f408e8c08cfcaf01e12a565874e71becc8372e16" Oct 30 19:12:39.245870 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:39.245827 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b75f08913ede59680b9233f408e8c08cfcaf01e12a565874e71becc8372e16"} err="failed to get container status \"35b75f08913ede59680b9233f408e8c08cfcaf01e12a565874e71becc8372e16\": rpc error: code = NotFound desc = could not find container \"35b75f08913ede59680b9233f408e8c08cfcaf01e12a565874e71becc8372e16\": container with ID starting with 35b75f08913ede59680b9233f408e8c08cfcaf01e12a565874e71becc8372e16 not found: ID does not exist" Oct 30 19:12:39.290513 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:39.290483 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g"] Oct 30 19:12:39.300641 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:39.300608 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5rm8g"] Oct 30 19:12:39.599778 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:12:39.599742 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:12:41.046818 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:12:41.046784 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10fa5b58-d1cb-447e-a1a6-f1c958569cb9" path="/var/lib/kubelet/pods/10fa5b58-d1cb-447e-a1a6-f1c958569cb9/volumes" Oct 30 19:13:09.599462 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:13:09.599415 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:13:39.599945 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:13:39.599840 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:14:07.093976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:07.093947 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:14:07.094874 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:07.094855 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:14:09.599741 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:14:09.599704 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:14:39.600331 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:14:39.600292 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:14:46.347485 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:46.347448 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-kdv7t"] Oct 30 19:14:46.348136 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:46.347936 5047 cpu_manager.go:414] "containerMap: removing container" podUID="10fa5b58-d1cb-447e-a1a6-f1c958569cb9" containerName="discovery" Oct 30 19:14:46.348136 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:46.347957 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10fa5b58-d1cb-447e-a1a6-f1c958569cb9" containerName="discovery" Oct 30 19:14:46.348136 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:46.348095 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="10fa5b58-d1cb-447e-a1a6-f1c958569cb9" containerName="discovery" Oct 30 19:14:46.351238 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:46.351217 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-kdv7t" Oct 30 19:14:46.366046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:46.366001 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-p9zfb\"" Oct 30 19:14:46.376294 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:46.376270 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 19:14:46.376294 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:46.376282 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 19:14:46.377433 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:46.377414 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-kdv7t"] Oct 30 19:14:46.527945 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:46.527882 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv4p4\" (UniqueName: \"kubernetes.io/projected/d7f89e1b-5ac7-46ce-9a5b-77a1ec2861b5-kube-api-access-jv4p4\") pod \"authorino-operator-68d6dd76dc-kdv7t\" (UID: \"d7f89e1b-5ac7-46ce-9a5b-77a1ec2861b5\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-kdv7t" Oct 30 19:14:46.628885 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:46.628781 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv4p4\" (UniqueName: \"kubernetes.io/projected/d7f89e1b-5ac7-46ce-9a5b-77a1ec2861b5-kube-api-access-jv4p4\") pod \"authorino-operator-68d6dd76dc-kdv7t\" (UID: \"d7f89e1b-5ac7-46ce-9a5b-77a1ec2861b5\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-kdv7t" Oct 30 19:14:46.650599 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:46.650570 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv4p4\" (UniqueName: \"kubernetes.io/projected/d7f89e1b-5ac7-46ce-9a5b-77a1ec2861b5-kube-api-access-jv4p4\") pod \"authorino-operator-68d6dd76dc-kdv7t\" (UID: \"d7f89e1b-5ac7-46ce-9a5b-77a1ec2861b5\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-kdv7t" Oct 30 19:14:46.663440 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:46.663404 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-kdv7t" Oct 30 19:14:46.967359 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:46.967271 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-kdv7t"] Oct 30 19:14:46.971661 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:14:46.971627 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f89e1b_5ac7_46ce_9a5b_77a1ec2861b5.slice/crio-feb4b0cb26817183b615b0938160cb8ce28ea1ce6e3b9a69ba71bb65eccb21ad WatchSource:0}: Error finding container feb4b0cb26817183b615b0938160cb8ce28ea1ce6e3b9a69ba71bb65eccb21ad: Status 404 returned error can't find the container with id feb4b0cb26817183b615b0938160cb8ce28ea1ce6e3b9a69ba71bb65eccb21ad Oct 30 19:14:47.761062 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:47.760995 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-kdv7t" event={"ID":"d7f89e1b-5ac7-46ce-9a5b-77a1ec2861b5","Type":"ContainerStarted","Data":"feb4b0cb26817183b615b0938160cb8ce28ea1ce6e3b9a69ba71bb65eccb21ad"} Oct 30 19:14:48.596033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:48.595977 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr"] Oct 30 19:14:48.605325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:48.605295 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" Oct 30 19:14:48.631704 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:48.631670 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lgg64\"" Oct 30 19:14:48.633532 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:48.633444 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr"] Oct 30 19:14:48.748345 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:48.748293 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f277h\" (UniqueName: \"kubernetes.io/projected/80bb4963-bcfa-44c8-b579-86535ade02e4-kube-api-access-f277h\") pod \"limitador-operator-controller-manager-856dd584c5-hj8kr\" (UID: \"80bb4963-bcfa-44c8-b579-86535ade02e4\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" Oct 30 19:14:48.849115 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:48.849027 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f277h\" (UniqueName: \"kubernetes.io/projected/80bb4963-bcfa-44c8-b579-86535ade02e4-kube-api-access-f277h\") pod \"limitador-operator-controller-manager-856dd584c5-hj8kr\" (UID: \"80bb4963-bcfa-44c8-b579-86535ade02e4\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" Oct 30 19:14:48.867565 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:48.867532 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f277h\" (UniqueName: \"kubernetes.io/projected/80bb4963-bcfa-44c8-b579-86535ade02e4-kube-api-access-f277h\") pod \"limitador-operator-controller-manager-856dd584c5-hj8kr\" (UID: \"80bb4963-bcfa-44c8-b579-86535ade02e4\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" Oct 30 19:14:48.921973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:48.921929 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" Oct 30 19:14:49.234619 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:49.234590 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr"] Oct 30 19:14:49.238080 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:14:49.238056 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80bb4963_bcfa_44c8_b579_86535ade02e4.slice/crio-67e7445aac1ec095f1b12cfefc8a89aa7ce740827caf80e2437340d778b23a92 WatchSource:0}: Error finding container 67e7445aac1ec095f1b12cfefc8a89aa7ce740827caf80e2437340d778b23a92: Status 404 returned error can't find the container with id 67e7445aac1ec095f1b12cfefc8a89aa7ce740827caf80e2437340d778b23a92 Oct 30 19:14:49.771249 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:49.771209 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" event={"ID":"80bb4963-bcfa-44c8-b579-86535ade02e4","Type":"ContainerStarted","Data":"67e7445aac1ec095f1b12cfefc8a89aa7ce740827caf80e2437340d778b23a92"} Oct 30 19:14:49.772727 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:49.772698 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-kdv7t" event={"ID":"d7f89e1b-5ac7-46ce-9a5b-77a1ec2861b5","Type":"ContainerStarted","Data":"025b8b29e565f9f37ba7b3f276032c8b51131806c935b1e4ab02733d2f99ed4b"} Oct 30 19:14:49.772852 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:49.772833 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-kdv7t" Oct 30 19:14:49.814152 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:49.814064 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-kdv7t" podStartSLOduration=1.6180495559999999 podStartE2EDuration="3.814047048s" totalImagesPullingTime="2.195997492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:49.812798425 +0000 UTC m=+642.887253819" watchObservedRunningTime="2026-10-30 19:14:49.814047048 +0000 UTC m=+642.888502404" Oct 30 19:14:50.778527 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:50.778491 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" event={"ID":"80bb4963-bcfa-44c8-b579-86535ade02e4","Type":"ContainerStarted","Data":"74759cad9e06c617084a614262e24aaa810199eef7ba3b95959846e66cef7e98"} Oct 30 19:14:50.778945 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:50.778576 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" Oct 30 19:14:50.817524 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:14:50.817461 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" podStartSLOduration=1.3797205479999999 podStartE2EDuration="2.817446107s" totalImagesPullingTime="1.437725559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:50.816187609 +0000 UTC m=+643.890642966" watchObservedRunningTime="2026-10-30 19:14:50.817446107 +0000 UTC m=+643.891901460" Oct 30 19:15:00.781173 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:00.781137 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-kdv7t" Oct 30 19:15:01.785714 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:01.785682 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" Oct 30 19:15:09.600167 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:15:09.600127 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:15:15.319075 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.319037 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn"] Oct 30 19:15:15.324320 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.324296 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" Oct 30 19:15:15.340297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.340263 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nzmp6\"" Oct 30 19:15:15.352426 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.352386 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn"] Oct 30 19:15:15.354612 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.354583 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr"] Oct 30 19:15:15.354853 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.354823 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" podUID="80bb4963-bcfa-44c8-b579-86535ade02e4" containerName="manager" containerID="cri-o://74759cad9e06c617084a614262e24aaa810199eef7ba3b95959846e66cef7e98" gracePeriod=2 Oct 30 19:15:15.355773 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.355648 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr"] Oct 30 19:15:15.372287 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.372259 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-nm5kw"] Oct 30 19:15:15.372769 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.372744 5047 cpu_manager.go:414] "containerMap: removing container" podUID="80bb4963-bcfa-44c8-b579-86535ade02e4" containerName="manager" Oct 30 19:15:15.372892 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.372771 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80bb4963-bcfa-44c8-b579-86535ade02e4" containerName="manager" Oct 30 19:15:15.372948 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.372937 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="80bb4963-bcfa-44c8-b579-86535ade02e4" containerName="manager" Oct 30 19:15:15.375630 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.375612 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nm5kw" Oct 30 19:15:15.378485 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:15:15.378456 5047 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-hj8kr\" is forbidden: User \"system:node:ip-10-0-140-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-235.ec2.internal' and this object" podUID="80bb4963-bcfa-44c8-b579-86535ade02e4" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" Oct 30 19:15:15.402629 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.402597 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-nm5kw"] Oct 30 19:15:15.408924 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.408893 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3972bcc-5a3d-4011-b2f2-f76817546b9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-k2zpn\" (UID: \"d3972bcc-5a3d-4011-b2f2-f76817546b9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" Oct 30 19:15:15.409108 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.408934 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mshtp\" (UniqueName: \"kubernetes.io/projected/d3972bcc-5a3d-4011-b2f2-f76817546b9a-kube-api-access-mshtp\") pod \"kuadrant-operator-controller-manager-6b6fb56846-k2zpn\" (UID: \"d3972bcc-5a3d-4011-b2f2-f76817546b9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" Oct 30 19:15:15.431135 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:15:15.431101 5047 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-hj8kr\" is forbidden: User \"system:node:ip-10-0-140-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-235.ec2.internal' and this object" podUID="80bb4963-bcfa-44c8-b579-86535ade02e4" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" Oct 30 19:15:15.509493 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.509458 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3972bcc-5a3d-4011-b2f2-f76817546b9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-k2zpn\" (UID: \"d3972bcc-5a3d-4011-b2f2-f76817546b9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" Oct 30 19:15:15.509673 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.509501 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mshtp\" (UniqueName: \"kubernetes.io/projected/d3972bcc-5a3d-4011-b2f2-f76817546b9a-kube-api-access-mshtp\") pod \"kuadrant-operator-controller-manager-6b6fb56846-k2zpn\" (UID: \"d3972bcc-5a3d-4011-b2f2-f76817546b9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" Oct 30 19:15:15.509673 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.509535 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skwhf\" (UniqueName: \"kubernetes.io/projected/369870ff-3389-4bb6-8b76-05a35b0cc02b-kube-api-access-skwhf\") pod \"limitador-operator-controller-manager-856dd584c5-nm5kw\" (UID: \"369870ff-3389-4bb6-8b76-05a35b0cc02b\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nm5kw" Oct 30 19:15:15.509851 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.509830 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3972bcc-5a3d-4011-b2f2-f76817546b9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-k2zpn\" (UID: \"d3972bcc-5a3d-4011-b2f2-f76817546b9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" Oct 30 19:15:15.528139 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.528095 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mshtp\" (UniqueName: \"kubernetes.io/projected/d3972bcc-5a3d-4011-b2f2-f76817546b9a-kube-api-access-mshtp\") pod \"kuadrant-operator-controller-manager-6b6fb56846-k2zpn\" (UID: \"d3972bcc-5a3d-4011-b2f2-f76817546b9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" Oct 30 19:15:15.608327 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.608299 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" Oct 30 19:15:15.610457 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.610435 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80bb4963-bcfa-44c8-b579-86535ade02e4-kube-api-access-f277h\" (UniqueName: \"kubernetes.io/projected/80bb4963-bcfa-44c8-b579-86535ade02e4-kube-api-access-f277h\") pod \"80bb4963-bcfa-44c8-b579-86535ade02e4\" (UID: \"80bb4963-bcfa-44c8-b579-86535ade02e4\") " Oct 30 19:15:15.610656 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.610639 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skwhf\" (UniqueName: \"kubernetes.io/projected/369870ff-3389-4bb6-8b76-05a35b0cc02b-kube-api-access-skwhf\") pod \"limitador-operator-controller-manager-856dd584c5-nm5kw\" (UID: \"369870ff-3389-4bb6-8b76-05a35b0cc02b\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nm5kw" Oct 30 19:15:15.612534 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.612491 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80bb4963-bcfa-44c8-b579-86535ade02e4-kube-api-access-f277h" pod "80bb4963-bcfa-44c8-b579-86535ade02e4" (UID: "80bb4963-bcfa-44c8-b579-86535ade02e4"). InnerVolumeSpecName "kube-api-access-f277h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:15.619590 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:15:15.619560 5047 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-hj8kr\" is forbidden: User \"system:node:ip-10-0-140-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-235.ec2.internal' and this object" podUID="80bb4963-bcfa-44c8-b579-86535ade02e4" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" Oct 30 19:15:15.637166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.637137 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" Oct 30 19:15:15.638190 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.638168 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skwhf\" (UniqueName: \"kubernetes.io/projected/369870ff-3389-4bb6-8b76-05a35b0cc02b-kube-api-access-skwhf\") pod \"limitador-operator-controller-manager-856dd584c5-nm5kw\" (UID: \"369870ff-3389-4bb6-8b76-05a35b0cc02b\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nm5kw" Oct 30 19:15:15.711619 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.711585 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f277h\" (UniqueName: \"kubernetes.io/projected/80bb4963-bcfa-44c8-b579-86535ade02e4-kube-api-access-f277h\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.729551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.729521 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nm5kw" Oct 30 19:15:15.950545 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.950464 5047 generic.go:356] "Generic (PLEG): container finished" podID="80bb4963-bcfa-44c8-b579-86535ade02e4" containerID="74759cad9e06c617084a614262e24aaa810199eef7ba3b95959846e66cef7e98" exitCode=0 Oct 30 19:15:15.950545 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.950508 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" Oct 30 19:15:15.950763 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.950564 5047 scope.go:122] "RemoveContainer" containerID="74759cad9e06c617084a614262e24aaa810199eef7ba3b95959846e66cef7e98" Oct 30 19:15:15.960133 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.960112 5047 scope.go:122] "RemoveContainer" containerID="74759cad9e06c617084a614262e24aaa810199eef7ba3b95959846e66cef7e98" Oct 30 19:15:15.960430 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:15:15.960409 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74759cad9e06c617084a614262e24aaa810199eef7ba3b95959846e66cef7e98\": container with ID starting with 74759cad9e06c617084a614262e24aaa810199eef7ba3b95959846e66cef7e98 not found: ID does not exist" containerID="74759cad9e06c617084a614262e24aaa810199eef7ba3b95959846e66cef7e98" Oct 30 19:15:15.960521 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.960437 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74759cad9e06c617084a614262e24aaa810199eef7ba3b95959846e66cef7e98"} err="failed to get container status \"74759cad9e06c617084a614262e24aaa810199eef7ba3b95959846e66cef7e98\": rpc error: code = NotFound desc = could not find container \"74759cad9e06c617084a614262e24aaa810199eef7ba3b95959846e66cef7e98\": container with ID starting with 74759cad9e06c617084a614262e24aaa810199eef7ba3b95959846e66cef7e98 not found: ID does not exist" Oct 30 19:15:15.960929 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:15.960906 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn"] Oct 30 19:15:15.961710 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:15:15.961687 5047 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-hj8kr\" is forbidden: User \"system:node:ip-10-0-140-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-235.ec2.internal' and this object" podUID="80bb4963-bcfa-44c8-b579-86535ade02e4" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" Oct 30 19:15:15.964084 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:15:15.964062 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3972bcc_5a3d_4011_b2f2_f76817546b9a.slice/crio-5fda8c09bea6309137fbf0117b59acc3b483fc0569c4f2023119413db21dafea WatchSource:0}: Error finding container 5fda8c09bea6309137fbf0117b59acc3b483fc0569c4f2023119413db21dafea: Status 404 returned error can't find the container with id 5fda8c09bea6309137fbf0117b59acc3b483fc0569c4f2023119413db21dafea Oct 30 19:15:15.973112 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:15:15.973087 5047 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-hj8kr\" is forbidden: User \"system:node:ip-10-0-140-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-235.ec2.internal' and this object" podUID="80bb4963-bcfa-44c8-b579-86535ade02e4" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" Oct 30 19:15:16.022122 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:16.022083 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-nm5kw"] Oct 30 19:15:16.026492 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:15:16.026465 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369870ff_3389_4bb6_8b76_05a35b0cc02b.slice/crio-b9c249af46d28b63083bfd66f0dbc68dd73f8bdea547fb3dff1b394eb262e82a WatchSource:0}: Error finding container b9c249af46d28b63083bfd66f0dbc68dd73f8bdea547fb3dff1b394eb262e82a: Status 404 returned error can't find the container with id b9c249af46d28b63083bfd66f0dbc68dd73f8bdea547fb3dff1b394eb262e82a Oct 30 19:15:16.963306 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:16.963269 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nm5kw" event={"ID":"369870ff-3389-4bb6-8b76-05a35b0cc02b","Type":"ContainerStarted","Data":"450ff4cbb7f73989c043eb072b07e7629df0283a05903108e5e63014107d541c"} Oct 30 19:15:16.963799 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:16.963334 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nm5kw" event={"ID":"369870ff-3389-4bb6-8b76-05a35b0cc02b","Type":"ContainerStarted","Data":"b9c249af46d28b63083bfd66f0dbc68dd73f8bdea547fb3dff1b394eb262e82a"} Oct 30 19:15:16.963799 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:16.963472 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nm5kw" Oct 30 19:15:16.965274 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:16.965242 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" event={"ID":"d3972bcc-5a3d-4011-b2f2-f76817546b9a","Type":"ContainerStarted","Data":"5fda8c09bea6309137fbf0117b59acc3b483fc0569c4f2023119413db21dafea"} Oct 30 19:15:16.974768 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:15:16.974731 5047 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-hj8kr\" is forbidden: User \"system:node:ip-10-0-140-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-235.ec2.internal' and this object" podUID="80bb4963-bcfa-44c8-b579-86535ade02e4" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" Oct 30 19:15:17.010147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:17.010080 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nm5kw" podStartSLOduration=2.010062969 podStartE2EDuration="2.010062969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:17.009584574 +0000 UTC m=+670.084039934" watchObservedRunningTime="2026-10-30 19:15:17.010062969 +0000 UTC m=+670.084518329" Oct 30 19:15:17.048713 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:17.048679 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80bb4963-bcfa-44c8-b579-86535ade02e4" path="/var/lib/kubelet/pods/80bb4963-bcfa-44c8-b579-86535ade02e4/volumes" Oct 30 19:15:17.054462 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:15:17.054426 5047 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-hj8kr\" is forbidden: User \"system:node:ip-10-0-140-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-235.ec2.internal' and this object" podUID="80bb4963-bcfa-44c8-b579-86535ade02e4" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-hj8kr" Oct 30 19:15:19.981924 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:19.981880 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" event={"ID":"d3972bcc-5a3d-4011-b2f2-f76817546b9a","Type":"ContainerStarted","Data":"4e6aba7a51ef86425e153128ba9f821e6f9b16fc3babfb8966901eb023e9f6e7"} Oct 30 19:15:19.982463 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:19.982026 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" Oct 30 19:15:20.038086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:20.038027 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" podStartSLOduration=1.503199994 podStartE2EDuration="5.037998491s" totalImagesPullingTime="3.534798497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:20.027361004 +0000 UTC m=+673.101816361" watchObservedRunningTime="2026-10-30 19:15:20.037998491 +0000 UTC m=+673.112453853" Oct 30 19:15:27.973767 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:27.973725 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-nm5kw" Oct 30 19:15:30.114645 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:30.114606 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn"] Oct 30 19:15:30.115107 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:30.114817 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" podUID="d3972bcc-5a3d-4011-b2f2-f76817546b9a" containerName="manager" containerID="cri-o://4e6aba7a51ef86425e153128ba9f821e6f9b16fc3babfb8966901eb023e9f6e7" gracePeriod=10 Oct 30 19:15:30.116813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:30.116789 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" Oct 30 19:15:30.358761 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:30.358738 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" Oct 30 19:15:30.434727 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:30.434628 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3972bcc-5a3d-4011-b2f2-f76817546b9a-kube-api-access-mshtp\" (UniqueName: \"kubernetes.io/projected/d3972bcc-5a3d-4011-b2f2-f76817546b9a-kube-api-access-mshtp\") pod \"d3972bcc-5a3d-4011-b2f2-f76817546b9a\" (UID: \"d3972bcc-5a3d-4011-b2f2-f76817546b9a\") " Oct 30 19:15:30.434727 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:30.434699 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3972bcc-5a3d-4011-b2f2-f76817546b9a-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3972bcc-5a3d-4011-b2f2-f76817546b9a-extensions-socket-volume\") pod \"d3972bcc-5a3d-4011-b2f2-f76817546b9a\" (UID: \"d3972bcc-5a3d-4011-b2f2-f76817546b9a\") " Oct 30 19:15:30.435237 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:30.435210 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3972bcc-5a3d-4011-b2f2-f76817546b9a-extensions-socket-volume" pod "d3972bcc-5a3d-4011-b2f2-f76817546b9a" (UID: "d3972bcc-5a3d-4011-b2f2-f76817546b9a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:30.436790 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:30.436759 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3972bcc-5a3d-4011-b2f2-f76817546b9a-kube-api-access-mshtp" pod "d3972bcc-5a3d-4011-b2f2-f76817546b9a" (UID: "d3972bcc-5a3d-4011-b2f2-f76817546b9a"). InnerVolumeSpecName "kube-api-access-mshtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:30.535346 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:30.535313 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mshtp\" (UniqueName: \"kubernetes.io/projected/d3972bcc-5a3d-4011-b2f2-f76817546b9a-kube-api-access-mshtp\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:15:30.535346 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:30.535341 5047 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3972bcc-5a3d-4011-b2f2-f76817546b9a-extensions-socket-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:15:31.038282 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:31.038244 5047 generic.go:356] "Generic (PLEG): container finished" podID="d3972bcc-5a3d-4011-b2f2-f76817546b9a" containerID="4e6aba7a51ef86425e153128ba9f821e6f9b16fc3babfb8966901eb023e9f6e7" exitCode=0 Oct 30 19:15:31.038282 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:31.038286 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" event={"ID":"d3972bcc-5a3d-4011-b2f2-f76817546b9a","Type":"ContainerDied","Data":"4e6aba7a51ef86425e153128ba9f821e6f9b16fc3babfb8966901eb023e9f6e7"} Oct 30 19:15:31.038509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:31.038295 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" Oct 30 19:15:31.038509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:31.038357 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn" event={"ID":"d3972bcc-5a3d-4011-b2f2-f76817546b9a","Type":"ContainerDied","Data":"5fda8c09bea6309137fbf0117b59acc3b483fc0569c4f2023119413db21dafea"} Oct 30 19:15:31.038509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:31.038379 5047 scope.go:122] "RemoveContainer" containerID="4e6aba7a51ef86425e153128ba9f821e6f9b16fc3babfb8966901eb023e9f6e7" Oct 30 19:15:31.047258 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:31.047225 5047 scope.go:122] "RemoveContainer" containerID="4e6aba7a51ef86425e153128ba9f821e6f9b16fc3babfb8966901eb023e9f6e7" Oct 30 19:15:31.047592 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:15:31.047562 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e6aba7a51ef86425e153128ba9f821e6f9b16fc3babfb8966901eb023e9f6e7\": container with ID starting with 4e6aba7a51ef86425e153128ba9f821e6f9b16fc3babfb8966901eb023e9f6e7 not found: ID does not exist" containerID="4e6aba7a51ef86425e153128ba9f821e6f9b16fc3babfb8966901eb023e9f6e7" Oct 30 19:15:31.047673 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:31.047615 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6aba7a51ef86425e153128ba9f821e6f9b16fc3babfb8966901eb023e9f6e7"} err="failed to get container status \"4e6aba7a51ef86425e153128ba9f821e6f9b16fc3babfb8966901eb023e9f6e7\": rpc error: code = NotFound desc = could not find container \"4e6aba7a51ef86425e153128ba9f821e6f9b16fc3babfb8966901eb023e9f6e7\": container with ID starting with 4e6aba7a51ef86425e153128ba9f821e6f9b16fc3babfb8966901eb023e9f6e7 not found: ID does not exist" Oct 30 19:15:31.101734 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:31.101692 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn"] Oct 30 19:15:31.111544 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:31.111513 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-k2zpn"] Oct 30 19:15:33.046881 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:15:33.046846 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3972bcc-5a3d-4011-b2f2-f76817546b9a" path="/var/lib/kubelet/pods/d3972bcc-5a3d-4011-b2f2-f76817546b9a/volumes" Oct 30 19:15:39.600665 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:15:39.600628 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:16:02.082459 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.082358 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cfs2r"] Oct 30 19:16:02.082994 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.082938 5047 cpu_manager.go:414] "containerMap: removing container" podUID="d3972bcc-5a3d-4011-b2f2-f76817546b9a" containerName="manager" Oct 30 19:16:02.082994 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.082955 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3972bcc-5a3d-4011-b2f2-f76817546b9a" containerName="manager" Oct 30 19:16:02.083097 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.083033 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3972bcc-5a3d-4011-b2f2-f76817546b9a" containerName="manager" Oct 30 19:16:02.086104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.086086 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" Oct 30 19:16:02.104654 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.104600 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cfs2r"] Oct 30 19:16:02.107233 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.107202 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwx9r\" (UniqueName: \"kubernetes.io/projected/0224dcc2-c993-416d-acca-33449c98f4fd-kube-api-access-cwx9r\") pod \"limitador-limitador-74dfcfb894-cfs2r\" (UID: \"0224dcc2-c993-416d-acca-33449c98f4fd\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" Oct 30 19:16:02.107382 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.107332 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0224dcc2-c993-416d-acca-33449c98f4fd-config-file\") pod \"limitador-limitador-74dfcfb894-cfs2r\" (UID: \"0224dcc2-c993-416d-acca-33449c98f4fd\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" Oct 30 19:16:02.112732 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.112712 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cpcvt\"" Oct 30 19:16:02.112732 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.112712 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 19:16:02.207789 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.207748 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0224dcc2-c993-416d-acca-33449c98f4fd-config-file\") pod \"limitador-limitador-74dfcfb894-cfs2r\" (UID: \"0224dcc2-c993-416d-acca-33449c98f4fd\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" Oct 30 19:16:02.207969 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.207832 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwx9r\" (UniqueName: \"kubernetes.io/projected/0224dcc2-c993-416d-acca-33449c98f4fd-kube-api-access-cwx9r\") pod \"limitador-limitador-74dfcfb894-cfs2r\" (UID: \"0224dcc2-c993-416d-acca-33449c98f4fd\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" Oct 30 19:16:02.208409 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.208382 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0224dcc2-c993-416d-acca-33449c98f4fd-config-file\") pod \"limitador-limitador-74dfcfb894-cfs2r\" (UID: \"0224dcc2-c993-416d-acca-33449c98f4fd\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" Oct 30 19:16:02.217548 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.217513 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cfs2r"] Oct 30 19:16:02.224955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.224920 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwx9r\" (UniqueName: \"kubernetes.io/projected/0224dcc2-c993-416d-acca-33449c98f4fd-kube-api-access-cwx9r\") pod \"limitador-limitador-74dfcfb894-cfs2r\" (UID: \"0224dcc2-c993-416d-acca-33449c98f4fd\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" Oct 30 19:16:02.399137 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.399034 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" Oct 30 19:16:02.700446 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.700370 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cfs2r"] Oct 30 19:16:02.704169 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:16:02.704138 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0224dcc2_c993_416d_acca_33449c98f4fd.slice/crio-bbaaf443aad454247478205f63f5b49398f9724e2823b863cb678c541519ddbe WatchSource:0}: Error finding container bbaaf443aad454247478205f63f5b49398f9724e2823b863cb678c541519ddbe: Status 404 returned error can't find the container with id bbaaf443aad454247478205f63f5b49398f9724e2823b863cb678c541519ddbe Oct 30 19:16:02.706110 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:02.706088 5047 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:16:03.179297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:03.179251 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" event={"ID":"0224dcc2-c993-416d-acca-33449c98f4fd","Type":"ContainerStarted","Data":"bbaaf443aad454247478205f63f5b49398f9724e2823b863cb678c541519ddbe"} Oct 30 19:16:09.209100 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:09.209051 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" event={"ID":"0224dcc2-c993-416d-acca-33449c98f4fd","Type":"ContainerStarted","Data":"2b8bafc4f5b3ff63fbf04273dc8b999ffedb1f66a192450bc5be0d2bc7033160"} Oct 30 19:16:09.209497 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:09.209140 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" Oct 30 19:16:09.245679 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:09.245624 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" podStartSLOduration=1.761321314 podStartE2EDuration="7.245610073s" totalImagesPullingTime="5.484288759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:09.244652513 +0000 UTC m=+722.319107872" watchObservedRunningTime="2026-10-30 19:16:09.245610073 +0000 UTC m=+722.320065423" Oct 30 19:16:09.600323 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:16:09.600286 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:16:16.310234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:16.310174 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cfs2r"] Oct 30 19:16:16.310812 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:16.310496 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" podUID="0224dcc2-c993-416d-acca-33449c98f4fd" containerName="limitador" containerID="cri-o://2b8bafc4f5b3ff63fbf04273dc8b999ffedb1f66a192450bc5be0d2bc7033160" gracePeriod=30 Oct 30 19:16:16.313434 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:16.313411 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" Oct 30 19:16:16.850697 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:16.850671 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" Oct 30 19:16:16.934747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:16.934670 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0224dcc2-c993-416d-acca-33449c98f4fd-kube-api-access-cwx9r\" (UniqueName: \"kubernetes.io/projected/0224dcc2-c993-416d-acca-33449c98f4fd-kube-api-access-cwx9r\") pod \"0224dcc2-c993-416d-acca-33449c98f4fd\" (UID: \"0224dcc2-c993-416d-acca-33449c98f4fd\") " Oct 30 19:16:16.934900 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:16.934797 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0224dcc2-c993-416d-acca-33449c98f4fd-config-file\" (UniqueName: \"kubernetes.io/configmap/0224dcc2-c993-416d-acca-33449c98f4fd-config-file\") pod \"0224dcc2-c993-416d-acca-33449c98f4fd\" (UID: \"0224dcc2-c993-416d-acca-33449c98f4fd\") " Oct 30 19:16:16.935178 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:16.935155 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0224dcc2-c993-416d-acca-33449c98f4fd-config-file" pod "0224dcc2-c993-416d-acca-33449c98f4fd" (UID: "0224dcc2-c993-416d-acca-33449c98f4fd"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:16.936823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:16.936803 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0224dcc2-c993-416d-acca-33449c98f4fd-kube-api-access-cwx9r" pod "0224dcc2-c993-416d-acca-33449c98f4fd" (UID: "0224dcc2-c993-416d-acca-33449c98f4fd"). InnerVolumeSpecName "kube-api-access-cwx9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:17.035814 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:17.035776 5047 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0224dcc2-c993-416d-acca-33449c98f4fd-config-file\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:16:17.035814 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:17.035807 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwx9r\" (UniqueName: \"kubernetes.io/projected/0224dcc2-c993-416d-acca-33449c98f4fd-kube-api-access-cwx9r\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:16:17.249512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:17.249424 5047 generic.go:356] "Generic (PLEG): container finished" podID="0224dcc2-c993-416d-acca-33449c98f4fd" containerID="2b8bafc4f5b3ff63fbf04273dc8b999ffedb1f66a192450bc5be0d2bc7033160" exitCode=0 Oct 30 19:16:17.249512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:17.249466 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" Oct 30 19:16:17.249696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:17.249469 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" event={"ID":"0224dcc2-c993-416d-acca-33449c98f4fd","Type":"ContainerDied","Data":"2b8bafc4f5b3ff63fbf04273dc8b999ffedb1f66a192450bc5be0d2bc7033160"} Oct 30 19:16:17.249696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:17.249592 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-cfs2r" event={"ID":"0224dcc2-c993-416d-acca-33449c98f4fd","Type":"ContainerDied","Data":"bbaaf443aad454247478205f63f5b49398f9724e2823b863cb678c541519ddbe"} Oct 30 19:16:17.249696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:17.249622 5047 scope.go:122] "RemoveContainer" containerID="2b8bafc4f5b3ff63fbf04273dc8b999ffedb1f66a192450bc5be0d2bc7033160" Oct 30 19:16:17.257856 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:17.257839 5047 scope.go:122] "RemoveContainer" containerID="2b8bafc4f5b3ff63fbf04273dc8b999ffedb1f66a192450bc5be0d2bc7033160" Oct 30 19:16:17.258105 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:16:17.258085 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b8bafc4f5b3ff63fbf04273dc8b999ffedb1f66a192450bc5be0d2bc7033160\": container with ID starting with 2b8bafc4f5b3ff63fbf04273dc8b999ffedb1f66a192450bc5be0d2bc7033160 not found: ID does not exist" containerID="2b8bafc4f5b3ff63fbf04273dc8b999ffedb1f66a192450bc5be0d2bc7033160" Oct 30 19:16:17.258153 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:17.258120 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8bafc4f5b3ff63fbf04273dc8b999ffedb1f66a192450bc5be0d2bc7033160"} err="failed to get container status \"2b8bafc4f5b3ff63fbf04273dc8b999ffedb1f66a192450bc5be0d2bc7033160\": rpc error: code = NotFound desc = could not find container \"2b8bafc4f5b3ff63fbf04273dc8b999ffedb1f66a192450bc5be0d2bc7033160\": container with ID starting with 2b8bafc4f5b3ff63fbf04273dc8b999ffedb1f66a192450bc5be0d2bc7033160 not found: ID does not exist" Oct 30 19:16:17.293658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:17.293629 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cfs2r"] Oct 30 19:16:17.303535 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:17.303511 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cfs2r"] Oct 30 19:16:19.046268 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:19.046236 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0224dcc2-c993-416d-acca-33449c98f4fd" path="/var/lib/kubelet/pods/0224dcc2-c993-416d-acca-33449c98f4fd/volumes" Oct 30 19:16:22.143409 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.143363 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-hmblk"] Oct 30 19:16:22.143896 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.143826 5047 cpu_manager.go:414] "containerMap: removing container" podUID="0224dcc2-c993-416d-acca-33449c98f4fd" containerName="limitador" Oct 30 19:16:22.143896 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.143846 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0224dcc2-c993-416d-acca-33449c98f4fd" containerName="limitador" Oct 30 19:16:22.144044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.143966 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="0224dcc2-c993-416d-acca-33449c98f4fd" containerName="limitador" Oct 30 19:16:22.147189 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.147167 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hmblk" Oct 30 19:16:22.159727 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.159706 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 19:16:22.169781 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.169704 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-vscgf\"" Oct 30 19:16:22.169781 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.169716 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:16:22.169781 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.169742 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:16:22.171558 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.171530 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-hmblk"] Oct 30 19:16:22.281243 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.281210 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/812cd787-806e-4ca5-8d92-e53ad0581e8c-data\") pod \"postgres-7cdbf6dc44-hmblk\" (UID: \"812cd787-806e-4ca5-8d92-e53ad0581e8c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hmblk" Oct 30 19:16:22.281243 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.281244 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7hgx\" (UniqueName: \"kubernetes.io/projected/812cd787-806e-4ca5-8d92-e53ad0581e8c-kube-api-access-c7hgx\") pod \"postgres-7cdbf6dc44-hmblk\" (UID: \"812cd787-806e-4ca5-8d92-e53ad0581e8c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hmblk" Oct 30 19:16:22.382234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.382189 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/812cd787-806e-4ca5-8d92-e53ad0581e8c-data\") pod \"postgres-7cdbf6dc44-hmblk\" (UID: \"812cd787-806e-4ca5-8d92-e53ad0581e8c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hmblk" Oct 30 19:16:22.382430 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.382252 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7hgx\" (UniqueName: \"kubernetes.io/projected/812cd787-806e-4ca5-8d92-e53ad0581e8c-kube-api-access-c7hgx\") pod \"postgres-7cdbf6dc44-hmblk\" (UID: \"812cd787-806e-4ca5-8d92-e53ad0581e8c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hmblk" Oct 30 19:16:22.382554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.382534 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/812cd787-806e-4ca5-8d92-e53ad0581e8c-data\") pod \"postgres-7cdbf6dc44-hmblk\" (UID: \"812cd787-806e-4ca5-8d92-e53ad0581e8c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hmblk" Oct 30 19:16:22.402194 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.402124 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7hgx\" (UniqueName: \"kubernetes.io/projected/812cd787-806e-4ca5-8d92-e53ad0581e8c-kube-api-access-c7hgx\") pod \"postgres-7cdbf6dc44-hmblk\" (UID: \"812cd787-806e-4ca5-8d92-e53ad0581e8c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hmblk" Oct 30 19:16:22.459245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.459206 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hmblk" Oct 30 19:16:22.751225 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:22.751142 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-hmblk"] Oct 30 19:16:22.754765 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:16:22.754735 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812cd787_806e_4ca5_8d92_e53ad0581e8c.slice/crio-3f4daad40a077a43b5642399435e23945fd03aabb8f9f982c2d14cace8fca23f WatchSource:0}: Error finding container 3f4daad40a077a43b5642399435e23945fd03aabb8f9f982c2d14cace8fca23f: Status 404 returned error can't find the container with id 3f4daad40a077a43b5642399435e23945fd03aabb8f9f982c2d14cace8fca23f Oct 30 19:16:23.277877 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:23.277843 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hmblk" event={"ID":"812cd787-806e-4ca5-8d92-e53ad0581e8c","Type":"ContainerStarted","Data":"3f4daad40a077a43b5642399435e23945fd03aabb8f9f982c2d14cace8fca23f"} Oct 30 19:16:28.305496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:28.305458 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hmblk" event={"ID":"812cd787-806e-4ca5-8d92-e53ad0581e8c","Type":"ContainerStarted","Data":"b62965229cca78a181136624b5e129f38944e06bb21a3a87e7229161db884346"} Oct 30 19:16:28.305895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:28.305570 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hmblk" Oct 30 19:16:28.341918 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:28.341842 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hmblk" podStartSLOduration=1.74002141 podStartE2EDuration="6.34182127s" totalImagesPullingTime="4.60179986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:28.340082771 +0000 UTC m=+741.414538129" watchObservedRunningTime="2026-10-30 19:16:28.34182127 +0000 UTC m=+741.416276630" Oct 30 19:16:34.339494 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:34.339465 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hmblk" Oct 30 19:16:39.600471 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:16:39.600424 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:16:54.479028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.478964 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-vfgdk"] Oct 30 19:16:54.482578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.482557 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-vfgdk" Oct 30 19:16:54.495897 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.495878 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 19:16:54.507191 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.507164 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 19:16:54.507191 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.507176 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-52jw9\"" Oct 30 19:16:54.507385 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.507184 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 19:16:54.508536 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.508514 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-vfgdk"] Oct 30 19:16:54.658632 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.658571 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-webhook-cert\") pod \"maas-controller-577f596567-vfgdk\" (UID: \"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015\") " pod="opendatahub/maas-controller-577f596567-vfgdk" Oct 30 19:16:54.658810 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.658642 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-metrics-certs\") pod \"maas-controller-577f596567-vfgdk\" (UID: \"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015\") " pod="opendatahub/maas-controller-577f596567-vfgdk" Oct 30 19:16:54.658810 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.658756 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28x2p\" (UniqueName: \"kubernetes.io/projected/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-kube-api-access-28x2p\") pod \"maas-controller-577f596567-vfgdk\" (UID: \"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015\") " pod="opendatahub/maas-controller-577f596567-vfgdk" Oct 30 19:16:54.759519 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.759430 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-webhook-cert\") pod \"maas-controller-577f596567-vfgdk\" (UID: \"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015\") " pod="opendatahub/maas-controller-577f596567-vfgdk" Oct 30 19:16:54.759519 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.759492 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-metrics-certs\") pod \"maas-controller-577f596567-vfgdk\" (UID: \"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015\") " pod="opendatahub/maas-controller-577f596567-vfgdk" Oct 30 19:16:54.759708 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.759567 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28x2p\" (UniqueName: \"kubernetes.io/projected/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-kube-api-access-28x2p\") pod \"maas-controller-577f596567-vfgdk\" (UID: \"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015\") " pod="opendatahub/maas-controller-577f596567-vfgdk" Oct 30 19:16:54.762062 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.762037 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-webhook-cert\") pod \"maas-controller-577f596567-vfgdk\" (UID: \"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015\") " pod="opendatahub/maas-controller-577f596567-vfgdk" Oct 30 19:16:54.762172 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.762104 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-metrics-certs\") pod \"maas-controller-577f596567-vfgdk\" (UID: \"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015\") " pod="opendatahub/maas-controller-577f596567-vfgdk" Oct 30 19:16:54.777111 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.777085 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28x2p\" (UniqueName: \"kubernetes.io/projected/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-kube-api-access-28x2p\") pod \"maas-controller-577f596567-vfgdk\" (UID: \"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015\") " pod="opendatahub/maas-controller-577f596567-vfgdk" Oct 30 19:16:54.794954 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.794921 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-vfgdk" Oct 30 19:16:54.822463 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.822422 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65cb454cc4-j4fbd"] Oct 30 19:16:54.825789 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.825769 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cb454cc4-j4fbd" Oct 30 19:16:54.848126 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.848089 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65cb454cc4-j4fbd"] Oct 30 19:16:54.961495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.961465 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffn6\" (UniqueName: \"kubernetes.io/projected/201d1cd3-34eb-4a4e-85c2-7198c93942c0-kube-api-access-6ffn6\") pod \"maas-controller-65cb454cc4-j4fbd\" (UID: \"201d1cd3-34eb-4a4e-85c2-7198c93942c0\") " pod="opendatahub/maas-controller-65cb454cc4-j4fbd" Oct 30 19:16:54.961664 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.961511 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/201d1cd3-34eb-4a4e-85c2-7198c93942c0-webhook-cert\") pod \"maas-controller-65cb454cc4-j4fbd\" (UID: \"201d1cd3-34eb-4a4e-85c2-7198c93942c0\") " pod="opendatahub/maas-controller-65cb454cc4-j4fbd" Oct 30 19:16:54.961664 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:54.961605 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/201d1cd3-34eb-4a4e-85c2-7198c93942c0-metrics-certs\") pod \"maas-controller-65cb454cc4-j4fbd\" (UID: \"201d1cd3-34eb-4a4e-85c2-7198c93942c0\") " pod="opendatahub/maas-controller-65cb454cc4-j4fbd" Oct 30 19:16:55.062883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:55.062805 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/201d1cd3-34eb-4a4e-85c2-7198c93942c0-metrics-certs\") pod \"maas-controller-65cb454cc4-j4fbd\" (UID: \"201d1cd3-34eb-4a4e-85c2-7198c93942c0\") " pod="opendatahub/maas-controller-65cb454cc4-j4fbd" Oct 30 19:16:55.063055 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:55.062877 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffn6\" (UniqueName: \"kubernetes.io/projected/201d1cd3-34eb-4a4e-85c2-7198c93942c0-kube-api-access-6ffn6\") pod \"maas-controller-65cb454cc4-j4fbd\" (UID: \"201d1cd3-34eb-4a4e-85c2-7198c93942c0\") " pod="opendatahub/maas-controller-65cb454cc4-j4fbd" Oct 30 19:16:55.063126 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:55.063104 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/201d1cd3-34eb-4a4e-85c2-7198c93942c0-webhook-cert\") pod \"maas-controller-65cb454cc4-j4fbd\" (UID: \"201d1cd3-34eb-4a4e-85c2-7198c93942c0\") " pod="opendatahub/maas-controller-65cb454cc4-j4fbd" Oct 30 19:16:55.065320 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:55.065291 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/201d1cd3-34eb-4a4e-85c2-7198c93942c0-webhook-cert\") pod \"maas-controller-65cb454cc4-j4fbd\" (UID: \"201d1cd3-34eb-4a4e-85c2-7198c93942c0\") " pod="opendatahub/maas-controller-65cb454cc4-j4fbd" Oct 30 19:16:55.065486 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:55.065386 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/201d1cd3-34eb-4a4e-85c2-7198c93942c0-metrics-certs\") pod \"maas-controller-65cb454cc4-j4fbd\" (UID: \"201d1cd3-34eb-4a4e-85c2-7198c93942c0\") " pod="opendatahub/maas-controller-65cb454cc4-j4fbd" Oct 30 19:16:55.079717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:55.079691 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffn6\" (UniqueName: \"kubernetes.io/projected/201d1cd3-34eb-4a4e-85c2-7198c93942c0-kube-api-access-6ffn6\") pod \"maas-controller-65cb454cc4-j4fbd\" (UID: \"201d1cd3-34eb-4a4e-85c2-7198c93942c0\") " pod="opendatahub/maas-controller-65cb454cc4-j4fbd" Oct 30 19:16:55.088340 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:55.088314 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-vfgdk"] Oct 30 19:16:55.091623 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:16:55.091596 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cecfdbf_0ebe_4f4a_b605_c085ba7e9015.slice/crio-9123974fc1ba4a92ba10bbbb393e93a1885300355c644b1f72623992dfa932af WatchSource:0}: Error finding container 9123974fc1ba4a92ba10bbbb393e93a1885300355c644b1f72623992dfa932af: Status 404 returned error can't find the container with id 9123974fc1ba4a92ba10bbbb393e93a1885300355c644b1f72623992dfa932af Oct 30 19:16:55.140417 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:55.140387 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cb454cc4-j4fbd" Oct 30 19:16:55.422952 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:55.422915 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-vfgdk" event={"ID":"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015","Type":"ContainerStarted","Data":"9123974fc1ba4a92ba10bbbb393e93a1885300355c644b1f72623992dfa932af"} Oct 30 19:16:55.432379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:55.432349 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65cb454cc4-j4fbd"] Oct 30 19:16:55.436551 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:16:55.436525 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201d1cd3_34eb_4a4e_85c2_7198c93942c0.slice/crio-d8c291318c3b781cb6000bdb9da6c3be6253ee15634b2d5a482c1045b506baee WatchSource:0}: Error finding container d8c291318c3b781cb6000bdb9da6c3be6253ee15634b2d5a482c1045b506baee: Status 404 returned error can't find the container with id d8c291318c3b781cb6000bdb9da6c3be6253ee15634b2d5a482c1045b506baee Oct 30 19:16:56.433338 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:56.433296 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cb454cc4-j4fbd" event={"ID":"201d1cd3-34eb-4a4e-85c2-7198c93942c0","Type":"ContainerStarted","Data":"d8c291318c3b781cb6000bdb9da6c3be6253ee15634b2d5a482c1045b506baee"} Oct 30 19:16:58.444630 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:58.444590 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cb454cc4-j4fbd" event={"ID":"201d1cd3-34eb-4a4e-85c2-7198c93942c0","Type":"ContainerStarted","Data":"38fa58e74c671b005d0965cb1100643d14834fb6402479057d0b41bb45028868"} Oct 30 19:16:58.445146 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:58.444758 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65cb454cc4-j4fbd" Oct 30 19:16:58.446111 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:58.446082 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-vfgdk" event={"ID":"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015","Type":"ContainerStarted","Data":"361fff55598335ab1cfadc6a37d16a9cbb81c3e4ab0f797ead5c8c466dba3ec7"} Oct 30 19:16:58.446255 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:58.446201 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-vfgdk" Oct 30 19:16:58.491634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:58.491574 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-65cb454cc4-j4fbd" podStartSLOduration=1.8573975919999999 podStartE2EDuration="4.49155718s" totalImagesPullingTime="2.634159588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:58.480882174 +0000 UTC m=+771.555337535" watchObservedRunningTime="2026-10-30 19:16:58.49155718 +0000 UTC m=+771.566012571" Oct 30 19:16:58.518087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:16:58.518027 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-vfgdk" podStartSLOduration=1.870574577 podStartE2EDuration="4.517993321s" totalImagesPullingTime="2.647418744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:58.516483722 +0000 UTC m=+771.590939078" watchObservedRunningTime="2026-10-30 19:16:58.517993321 +0000 UTC m=+771.592448682" Oct 30 19:17:01.374371 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.374334 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl"] Oct 30 19:17:01.377907 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.377889 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" Oct 30 19:17:01.392849 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.392820 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 19:17:01.394482 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.394455 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl"] Oct 30 19:17:01.403086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.403057 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 19:17:01.403086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.403057 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 19:17:01.403317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.403263 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-dvbmf\"" Oct 30 19:17:01.426636 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.426606 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a703457c-cd6f-4b39-b8db-e7c3c419e72d-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-vz8vl\" (UID: \"a703457c-cd6f-4b39-b8db-e7c3c419e72d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" Oct 30 19:17:01.426861 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.426667 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq7dr\" (UniqueName: \"kubernetes.io/projected/a703457c-cd6f-4b39-b8db-e7c3c419e72d-kube-api-access-cq7dr\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-vz8vl\" (UID: \"a703457c-cd6f-4b39-b8db-e7c3c419e72d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" Oct 30 19:17:01.426861 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.426820 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a703457c-cd6f-4b39-b8db-e7c3c419e72d-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-vz8vl\" (UID: \"a703457c-cd6f-4b39-b8db-e7c3c419e72d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" Oct 30 19:17:01.426986 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.426893 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a703457c-cd6f-4b39-b8db-e7c3c419e72d-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-vz8vl\" (UID: \"a703457c-cd6f-4b39-b8db-e7c3c419e72d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" Oct 30 19:17:01.527442 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.527396 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq7dr\" (UniqueName: \"kubernetes.io/projected/a703457c-cd6f-4b39-b8db-e7c3c419e72d-kube-api-access-cq7dr\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-vz8vl\" (UID: \"a703457c-cd6f-4b39-b8db-e7c3c419e72d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" Oct 30 19:17:01.527667 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.527484 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a703457c-cd6f-4b39-b8db-e7c3c419e72d-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-vz8vl\" (UID: \"a703457c-cd6f-4b39-b8db-e7c3c419e72d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" Oct 30 19:17:01.527667 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.527517 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a703457c-cd6f-4b39-b8db-e7c3c419e72d-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-vz8vl\" (UID: \"a703457c-cd6f-4b39-b8db-e7c3c419e72d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" Oct 30 19:17:01.527667 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.527581 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a703457c-cd6f-4b39-b8db-e7c3c419e72d-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-vz8vl\" (UID: \"a703457c-cd6f-4b39-b8db-e7c3c419e72d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" Oct 30 19:17:01.528015 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:17:01.527986 5047 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 30 19:17:01.528141 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:17:01.528118 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a703457c-cd6f-4b39-b8db-e7c3c419e72d-tls-cert podName:a703457c-cd6f-4b39-b8db-e7c3c419e72d nodeName:}" failed. No retries permitted until 2026-10-30 19:17:02.02809488 +0000 UTC m=+775.102550234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/a703457c-cd6f-4b39-b8db-e7c3c419e72d-tls-cert") pod "usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" (UID: "a703457c-cd6f-4b39-b8db-e7c3c419e72d") : secret "usage-logs-tenancy-proxy-tls" not found Oct 30 19:17:01.528392 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.528368 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a703457c-cd6f-4b39-b8db-e7c3c419e72d-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-vz8vl\" (UID: \"a703457c-cd6f-4b39-b8db-e7c3c419e72d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" Oct 30 19:17:01.528814 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.528792 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a703457c-cd6f-4b39-b8db-e7c3c419e72d-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-vz8vl\" (UID: \"a703457c-cd6f-4b39-b8db-e7c3c419e72d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" Oct 30 19:17:01.545406 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:01.545375 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq7dr\" (UniqueName: \"kubernetes.io/projected/a703457c-cd6f-4b39-b8db-e7c3c419e72d-kube-api-access-cq7dr\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-vz8vl\" (UID: \"a703457c-cd6f-4b39-b8db-e7c3c419e72d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" Oct 30 19:17:02.031612 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:02.031571 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a703457c-cd6f-4b39-b8db-e7c3c419e72d-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-vz8vl\" (UID: \"a703457c-cd6f-4b39-b8db-e7c3c419e72d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" Oct 30 19:17:02.034023 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:02.033982 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a703457c-cd6f-4b39-b8db-e7c3c419e72d-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-vz8vl\" (UID: \"a703457c-cd6f-4b39-b8db-e7c3c419e72d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" Oct 30 19:17:02.290546 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:02.290448 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" Oct 30 19:17:02.582521 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:02.582492 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl"] Oct 30 19:17:02.585939 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:17:02.585907 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda703457c_cd6f_4b39_b8db_e7c3c419e72d.slice/crio-748a9c63bd5824fb1e540509cf27788b572aa0dc7a2851b920d2f0267de5a615 WatchSource:0}: Error finding container 748a9c63bd5824fb1e540509cf27788b572aa0dc7a2851b920d2f0267de5a615: Status 404 returned error can't find the container with id 748a9c63bd5824fb1e540509cf27788b572aa0dc7a2851b920d2f0267de5a615 Oct 30 19:17:03.469329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:03.469297 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" event={"ID":"a703457c-cd6f-4b39-b8db-e7c3c419e72d","Type":"ContainerStarted","Data":"748a9c63bd5824fb1e540509cf27788b572aa0dc7a2851b920d2f0267de5a615"} Oct 30 19:17:09.452525 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:09.452487 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-vfgdk" Oct 30 19:17:09.452997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:09.452576 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65cb454cc4-j4fbd" Oct 30 19:17:09.515333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:09.515299 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-vfgdk"] Oct 30 19:17:09.515555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:09.515511 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-vfgdk" podUID="5cecfdbf-0ebe-4f4a-b605-c085ba7e9015" containerName="manager" containerID="cri-o://361fff55598335ab1cfadc6a37d16a9cbb81c3e4ab0f797ead5c8c466dba3ec7" gracePeriod=30 Oct 30 19:17:09.603962 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:17:09.603921 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:17:09.787065 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:09.787034 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-vfgdk" Oct 30 19:17:09.909131 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:09.909095 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-metrics-certs\") pod \"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015\" (UID: \"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015\") " Oct 30 19:17:09.909309 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:09.909185 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-kube-api-access-28x2p\" (UniqueName: \"kubernetes.io/projected/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-kube-api-access-28x2p\") pod \"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015\" (UID: \"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015\") " Oct 30 19:17:09.909309 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:09.909218 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-webhook-cert\") pod \"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015\" (UID: \"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015\") " Oct 30 19:17:09.911754 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:09.911722 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-metrics-certs" pod "5cecfdbf-0ebe-4f4a-b605-c085ba7e9015" (UID: "5cecfdbf-0ebe-4f4a-b605-c085ba7e9015"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:09.911909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:09.911756 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-kube-api-access-28x2p" pod "5cecfdbf-0ebe-4f4a-b605-c085ba7e9015" (UID: "5cecfdbf-0ebe-4f4a-b605-c085ba7e9015"). InnerVolumeSpecName "kube-api-access-28x2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:09.911909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:09.911793 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-webhook-cert" pod "5cecfdbf-0ebe-4f4a-b605-c085ba7e9015" (UID: "5cecfdbf-0ebe-4f4a-b605-c085ba7e9015"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:10.010838 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:10.010796 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:17:10.010838 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:10.010835 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28x2p\" (UniqueName: \"kubernetes.io/projected/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-kube-api-access-28x2p\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:17:10.011106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:10.010850 5047 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015-webhook-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:17:10.512552 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:10.512507 5047 generic.go:356] "Generic (PLEG): container finished" podID="5cecfdbf-0ebe-4f4a-b605-c085ba7e9015" containerID="361fff55598335ab1cfadc6a37d16a9cbb81c3e4ab0f797ead5c8c466dba3ec7" exitCode=0 Oct 30 19:17:10.512991 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:10.512568 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-vfgdk" Oct 30 19:17:10.512991 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:10.512574 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-vfgdk" event={"ID":"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015","Type":"ContainerDied","Data":"361fff55598335ab1cfadc6a37d16a9cbb81c3e4ab0f797ead5c8c466dba3ec7"} Oct 30 19:17:10.512991 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:10.512619 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-vfgdk" event={"ID":"5cecfdbf-0ebe-4f4a-b605-c085ba7e9015","Type":"ContainerDied","Data":"9123974fc1ba4a92ba10bbbb393e93a1885300355c644b1f72623992dfa932af"} Oct 30 19:17:10.512991 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:10.512641 5047 scope.go:122] "RemoveContainer" containerID="361fff55598335ab1cfadc6a37d16a9cbb81c3e4ab0f797ead5c8c466dba3ec7" Oct 30 19:17:10.525034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:10.524967 5047 scope.go:122] "RemoveContainer" containerID="361fff55598335ab1cfadc6a37d16a9cbb81c3e4ab0f797ead5c8c466dba3ec7" Oct 30 19:17:10.525416 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:17:10.525381 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361fff55598335ab1cfadc6a37d16a9cbb81c3e4ab0f797ead5c8c466dba3ec7\": container with ID starting with 361fff55598335ab1cfadc6a37d16a9cbb81c3e4ab0f797ead5c8c466dba3ec7 not found: ID does not exist" containerID="361fff55598335ab1cfadc6a37d16a9cbb81c3e4ab0f797ead5c8c466dba3ec7" Oct 30 19:17:10.525691 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:10.525664 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361fff55598335ab1cfadc6a37d16a9cbb81c3e4ab0f797ead5c8c466dba3ec7"} err="failed to get container status \"361fff55598335ab1cfadc6a37d16a9cbb81c3e4ab0f797ead5c8c466dba3ec7\": rpc error: code = NotFound desc = could not find container \"361fff55598335ab1cfadc6a37d16a9cbb81c3e4ab0f797ead5c8c466dba3ec7\": container with ID starting with 361fff55598335ab1cfadc6a37d16a9cbb81c3e4ab0f797ead5c8c466dba3ec7 not found: ID does not exist" Oct 30 19:17:10.558570 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:10.558536 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-vfgdk"] Oct 30 19:17:10.568207 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:10.568169 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-vfgdk"] Oct 30 19:17:11.048148 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:11.048112 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cecfdbf-0ebe-4f4a-b605-c085ba7e9015" path="/var/lib/kubelet/pods/5cecfdbf-0ebe-4f4a-b605-c085ba7e9015/volumes" Oct 30 19:17:14.533917 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:14.533868 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" event={"ID":"a703457c-cd6f-4b39-b8db-e7c3c419e72d","Type":"ContainerStarted","Data":"b7c030e1a1e48e070e8ea35b3b81c7ed3312d3861279861dc4570141f6fe26f3"} Oct 30 19:17:14.534455 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:14.533956 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" Oct 30 19:17:14.569911 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:14.569850 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" podStartSLOduration=2.658227295 podStartE2EDuration="13.569834603s" totalImagesPullingTime="10.911607308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:14.567893477 +0000 UTC m=+787.642348835" watchObservedRunningTime="2026-10-30 19:17:14.569834603 +0000 UTC m=+787.644289961" Oct 30 19:17:25.544589 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:25.544555 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-vz8vl" Oct 30 19:17:31.937246 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:31.937209 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c"] Oct 30 19:17:31.937682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:31.937611 5047 cpu_manager.go:414] "containerMap: removing container" podUID="5cecfdbf-0ebe-4f4a-b605-c085ba7e9015" containerName="manager" Oct 30 19:17:31.937682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:31.937624 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cecfdbf-0ebe-4f4a-b605-c085ba7e9015" containerName="manager" Oct 30 19:17:31.937773 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:31.937687 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cecfdbf-0ebe-4f4a-b605-c085ba7e9015" containerName="manager" Oct 30 19:17:31.943182 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:31.943160 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 19:17:31.958891 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:31.958865 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c"] Oct 30 19:17:31.967503 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:31.967475 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:17:31.967503 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:31.967492 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r84xv\"" Oct 30 19:17:31.970230 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:31.970202 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 19:17:32.101951 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:32.101918 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c62k\" (UniqueName: \"kubernetes.io/projected/b0d053db-3c2e-4701-b045-68a75abeaa89-kube-api-access-2c62k\") pod \"maas-api-5c48877c75-jlg7c\" (UID: \"b0d053db-3c2e-4701-b045-68a75abeaa89\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 19:17:32.102146 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:32.101970 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0d053db-3c2e-4701-b045-68a75abeaa89-metrics-certs\") pod \"maas-api-5c48877c75-jlg7c\" (UID: \"b0d053db-3c2e-4701-b045-68a75abeaa89\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 19:17:32.102146 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:32.101994 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0d053db-3c2e-4701-b045-68a75abeaa89-maas-api-tls\") pod \"maas-api-5c48877c75-jlg7c\" (UID: \"b0d053db-3c2e-4701-b045-68a75abeaa89\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 19:17:32.102225 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:32.102136 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0d053db-3c2e-4701-b045-68a75abeaa89-openshift-service-ca\") pod \"maas-api-5c48877c75-jlg7c\" (UID: \"b0d053db-3c2e-4701-b045-68a75abeaa89\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 19:17:32.203415 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:32.203332 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c62k\" (UniqueName: \"kubernetes.io/projected/b0d053db-3c2e-4701-b045-68a75abeaa89-kube-api-access-2c62k\") pod \"maas-api-5c48877c75-jlg7c\" (UID: \"b0d053db-3c2e-4701-b045-68a75abeaa89\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 19:17:32.203415 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:32.203385 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0d053db-3c2e-4701-b045-68a75abeaa89-metrics-certs\") pod \"maas-api-5c48877c75-jlg7c\" (UID: \"b0d053db-3c2e-4701-b045-68a75abeaa89\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 19:17:32.203652 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:32.203416 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0d053db-3c2e-4701-b045-68a75abeaa89-maas-api-tls\") pod \"maas-api-5c48877c75-jlg7c\" (UID: \"b0d053db-3c2e-4701-b045-68a75abeaa89\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 19:17:32.203652 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:32.203451 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0d053db-3c2e-4701-b045-68a75abeaa89-openshift-service-ca\") pod \"maas-api-5c48877c75-jlg7c\" (UID: \"b0d053db-3c2e-4701-b045-68a75abeaa89\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 19:17:32.204252 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:32.204228 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0d053db-3c2e-4701-b045-68a75abeaa89-openshift-service-ca\") pod \"maas-api-5c48877c75-jlg7c\" (UID: \"b0d053db-3c2e-4701-b045-68a75abeaa89\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 19:17:32.206064 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:32.206040 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0d053db-3c2e-4701-b045-68a75abeaa89-maas-api-tls\") pod \"maas-api-5c48877c75-jlg7c\" (UID: \"b0d053db-3c2e-4701-b045-68a75abeaa89\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 19:17:32.206287 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:32.206258 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0d053db-3c2e-4701-b045-68a75abeaa89-metrics-certs\") pod \"maas-api-5c48877c75-jlg7c\" (UID: \"b0d053db-3c2e-4701-b045-68a75abeaa89\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 19:17:32.219955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:32.219925 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c62k\" (UniqueName: \"kubernetes.io/projected/b0d053db-3c2e-4701-b045-68a75abeaa89-kube-api-access-2c62k\") pod \"maas-api-5c48877c75-jlg7c\" (UID: \"b0d053db-3c2e-4701-b045-68a75abeaa89\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 19:17:32.255470 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:32.255436 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 19:17:32.548237 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:32.548146 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c"] Oct 30 19:17:32.552301 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:17:32.552266 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d053db_3c2e_4701_b045_68a75abeaa89.slice/crio-ef4689abcc9451f296ae9c37641fe16095c1338075b1069417e0c19dc5ec2228 WatchSource:0}: Error finding container ef4689abcc9451f296ae9c37641fe16095c1338075b1069417e0c19dc5ec2228: Status 404 returned error can't find the container with id ef4689abcc9451f296ae9c37641fe16095c1338075b1069417e0c19dc5ec2228 Oct 30 19:17:32.616375 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:32.616339 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" event={"ID":"b0d053db-3c2e-4701-b045-68a75abeaa89","Type":"ContainerStarted","Data":"ef4689abcc9451f296ae9c37641fe16095c1338075b1069417e0c19dc5ec2228"} Oct 30 19:17:34.627480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:34.627440 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" event={"ID":"b0d053db-3c2e-4701-b045-68a75abeaa89","Type":"ContainerStarted","Data":"100b47dfb8ebd1de2e3239f1c18206449c7e7b4af5676cfbd3693a47bd998471"} Oct 30 19:17:34.627871 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:34.627573 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 19:17:34.663526 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:34.663437 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" podStartSLOduration=2.204820472 podStartE2EDuration="3.663419608s" totalImagesPullingTime="1.458599136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:34.661807476 +0000 UTC m=+807.736262838" watchObservedRunningTime="2026-10-30 19:17:34.663419608 +0000 UTC m=+807.737874968" Oct 30 19:17:39.600441 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:17:39.600404 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:17:40.638091 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:17:40.638061 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 19:18:09.601250 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:18:09.601212 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:18:39.600725 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:18:39.600608 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:18:53.559370 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.559330 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg"] Oct 30 19:18:53.561817 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.561791 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.574577 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.574548 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 19:18:53.574577 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.574553 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 19:18:53.584543 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.584496 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 19:18:53.584721 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.584549 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-48lvl\"" Oct 30 19:18:53.586276 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.586255 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg"] Oct 30 19:18:53.610721 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.610688 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.610854 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.610729 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt8zf\" (UniqueName: \"kubernetes.io/projected/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-kube-api-access-kt8zf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.610854 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.610815 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.610854 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.610845 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.610953 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.610873 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.610953 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.610935 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.712091 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.712036 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.712091 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.712096 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt8zf\" (UniqueName: \"kubernetes.io/projected/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-kube-api-access-kt8zf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.712350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.712128 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.712350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.712144 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.712350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.712168 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.712350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.712183 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.712564 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.712527 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.712625 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.712587 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.712683 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.712639 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.714407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.714384 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.714755 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.714735 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.731778 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.731751 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt8zf\" (UniqueName: \"kubernetes.io/projected/d6cf3827-a5fe-48e4-900a-1f1633ff69ca-kube-api-access-kt8zf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg\" (UID: \"d6cf3827-a5fe-48e4-900a-1f1633ff69ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:53.878751 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:53.878707 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:54.175885 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:54.175806 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg"] Oct 30 19:18:54.179880 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:18:54.179847 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6cf3827_a5fe_48e4_900a_1f1633ff69ca.slice/crio-4d4d91d23f6343019b026d400967c107d16a460bf6a57cb64cc63fbc03b43eee WatchSource:0}: Error finding container 4d4d91d23f6343019b026d400967c107d16a460bf6a57cb64cc63fbc03b43eee: Status 404 returned error can't find the container with id 4d4d91d23f6343019b026d400967c107d16a460bf6a57cb64cc63fbc03b43eee Oct 30 19:18:54.987406 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:54.987359 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" event={"ID":"d6cf3827-a5fe-48e4-900a-1f1633ff69ca","Type":"ContainerStarted","Data":"4d4d91d23f6343019b026d400967c107d16a460bf6a57cb64cc63fbc03b43eee"} Oct 30 19:18:55.993444 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:55.993405 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" event={"ID":"d6cf3827-a5fe-48e4-900a-1f1633ff69ca","Type":"ContainerStarted","Data":"07571801be57a863d06e79beda161a7d4699dc3289ec4297fc176a2c6df2f4b8"} Oct 30 19:18:56.031067 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:56.030978 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" podStartSLOduration=2.012019082 podStartE2EDuration="3.030957118s" totalImagesPullingTime="1.018938036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:56.02929666 +0000 UTC m=+889.103752022" watchObservedRunningTime="2026-10-30 19:18:56.030957118 +0000 UTC m=+889.105412548" Oct 30 19:18:56.878848 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:56.878808 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:18:58.048124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.048082 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn"] Oct 30 19:18:58.051227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.051203 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.063509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.063487 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 19:18:58.074993 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.074966 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn"] Oct 30 19:18:58.152767 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.152727 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c276eb19-3a26-4138-9545-d465361783f6-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.152959 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.152801 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c276eb19-3a26-4138-9545-d465361783f6-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.152959 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.152895 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c276eb19-3a26-4138-9545-d465361783f6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.152959 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.152925 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs7d5\" (UniqueName: \"kubernetes.io/projected/c276eb19-3a26-4138-9545-d465361783f6-kube-api-access-zs7d5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.153119 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.152960 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c276eb19-3a26-4138-9545-d465361783f6-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.153119 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.153110 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c276eb19-3a26-4138-9545-d465361783f6-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.254383 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.254348 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs7d5\" (UniqueName: \"kubernetes.io/projected/c276eb19-3a26-4138-9545-d465361783f6-kube-api-access-zs7d5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.254383 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.254391 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c276eb19-3a26-4138-9545-d465361783f6-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.254640 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.254421 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c276eb19-3a26-4138-9545-d465361783f6-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.254640 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.254456 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c276eb19-3a26-4138-9545-d465361783f6-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.254640 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.254491 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c276eb19-3a26-4138-9545-d465361783f6-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.254640 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.254519 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c276eb19-3a26-4138-9545-d465361783f6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.254912 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.254883 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c276eb19-3a26-4138-9545-d465361783f6-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.254912 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.254898 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c276eb19-3a26-4138-9545-d465361783f6-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.255137 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.254931 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c276eb19-3a26-4138-9545-d465361783f6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.256747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.256720 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c276eb19-3a26-4138-9545-d465361783f6-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.256916 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.256899 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c276eb19-3a26-4138-9545-d465361783f6-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.270751 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.270726 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs7d5\" (UniqueName: \"kubernetes.io/projected/c276eb19-3a26-4138-9545-d465361783f6-kube-api-access-zs7d5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5bffn\" (UID: \"c276eb19-3a26-4138-9545-d465361783f6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.363529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.363489 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:18:58.657936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:58.657856 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn"] Oct 30 19:18:58.661536 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:18:58.661507 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc276eb19_3a26_4138_9545_d465361783f6.slice/crio-ae1b55c2bea547264bf1e48b97348751ce2a60163c9f9c8c1b8d1db64ca039b3 WatchSource:0}: Error finding container ae1b55c2bea547264bf1e48b97348751ce2a60163c9f9c8c1b8d1db64ca039b3: Status 404 returned error can't find the container with id ae1b55c2bea547264bf1e48b97348751ce2a60163c9f9c8c1b8d1db64ca039b3 Oct 30 19:18:59.008661 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:59.008620 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" event={"ID":"c276eb19-3a26-4138-9545-d465361783f6","Type":"ContainerStarted","Data":"95decaaa4f1fbdde32769c61a38f3e09ac354b78093b9892f460b691cfbf1ffc"} Oct 30 19:18:59.008824 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:59.008686 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" event={"ID":"c276eb19-3a26-4138-9545-d465361783f6","Type":"ContainerStarted","Data":"ae1b55c2bea547264bf1e48b97348751ce2a60163c9f9c8c1b8d1db64ca039b3"} Oct 30 19:18:59.044485 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:59.044439 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" podStartSLOduration=0.82821754 podStartE2EDuration="1.044423396s" totalImagesPullingTime="216.205856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:59.042272732 +0000 UTC m=+892.116728091" watchObservedRunningTime="2026-10-30 19:18:59.044423396 +0000 UTC m=+892.118878755" Oct 30 19:18:59.364130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:18:59.364101 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:19:03.879852 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:03.879802 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:19:03.892653 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:03.892624 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:19:04.043399 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.043365 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" Oct 30 19:19:04.495831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.495798 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g"] Oct 30 19:19:04.501866 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.501838 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.519707 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.519677 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g"] Oct 30 19:19:04.524624 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.524593 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 19:19:04.611637 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.611597 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.611811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.611647 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.611811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.611764 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgcn2\" (UniqueName: \"kubernetes.io/projected/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-kube-api-access-hgcn2\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.611811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.611798 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.611917 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.611823 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.611956 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.611933 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.713359 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.713316 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.713551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.713369 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.713551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.713396 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.713551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.713427 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgcn2\" (UniqueName: \"kubernetes.io/projected/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-kube-api-access-hgcn2\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.713551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.713458 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.713551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.713478 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.713893 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.713859 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.714053 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.713930 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.714053 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.713958 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.715684 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.715659 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.715944 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.715923 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.730719 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.730688 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgcn2\" (UniqueName: \"kubernetes.io/projected/1b105e8c-2f71-484d-b6fa-454fdc1dc3e2-kube-api-access-hgcn2\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g\" (UID: \"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:04.818192 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:04.818109 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:05.112416 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:05.112384 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g"] Oct 30 19:19:05.116029 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:19:05.115986 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b105e8c_2f71_484d_b6fa_454fdc1dc3e2.slice/crio-45db3b737810d0cc8db73edc3083ec3305b003714c823a3447760f845fec807a WatchSource:0}: Error finding container 45db3b737810d0cc8db73edc3083ec3305b003714c823a3447760f845fec807a: Status 404 returned error can't find the container with id 45db3b737810d0cc8db73edc3083ec3305b003714c823a3447760f845fec807a Oct 30 19:19:06.042531 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:06.042492 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" event={"ID":"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2","Type":"ContainerStarted","Data":"b37b43f934dd642654a31e43ed64b3d5946b5bc0de58ab37a124b201938b465f"} Oct 30 19:19:06.042747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:06.042579 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" event={"ID":"1b105e8c-2f71-484d-b6fa-454fdc1dc3e2","Type":"ContainerStarted","Data":"45db3b737810d0cc8db73edc3083ec3305b003714c823a3447760f845fec807a"} Oct 30 19:19:06.078080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:06.078027 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" podStartSLOduration=1.8315950330000002 podStartE2EDuration="2.077995105s" totalImagesPullingTime="246.400072ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:06.076090548 +0000 UTC m=+899.150545908" watchObservedRunningTime="2026-10-30 19:19:06.077995105 +0000 UTC m=+899.152450466" Oct 30 19:19:06.819140 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:06.819082 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:07.126198 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:07.126165 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:19:07.127077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:07.126948 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:19:08.364463 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:08.364426 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:19:08.377185 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:08.377151 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:19:09.067508 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:09.067475 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5bffn" Oct 30 19:19:09.599909 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:19:09.599865 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:19:14.819112 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:14.819070 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:14.831220 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:14.831194 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:15.095579 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:15.095550 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" Oct 30 19:19:17.906230 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:17.906190 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26"] Oct 30 19:19:17.911272 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:17.911249 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:17.925136 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:17.925109 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z"] Oct 30 19:19:17.928830 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:17.928811 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:17.933558 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:17.933527 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bw5jw\"" Oct 30 19:19:17.935399 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:17.935376 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26"] Oct 30 19:19:17.951515 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:17.951483 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z"] Oct 30 19:19:18.039142 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039104 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.039329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039147 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.039329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039238 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.039329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039284 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwlr4\" (UniqueName: \"kubernetes.io/projected/c724ef66-462d-429a-8b12-c6b9d7f90a1e-kube-api-access-fwlr4\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.039329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039303 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.039495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039354 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.039495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039389 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.039495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039407 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.039495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039458 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh74z\" (UniqueName: \"kubernetes.io/projected/3824e276-5eec-4cea-a06b-c81cc9853ed4-kube-api-access-qh74z\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.039495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039486 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.039710 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039503 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.039710 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039521 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.039710 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039537 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.039710 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039636 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.039710 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039658 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.039710 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039675 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.039911 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039742 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.039911 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.039763 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3824e276-5eec-4cea-a06b-c81cc9853ed4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.140707 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.140669 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.140908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.140724 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwlr4\" (UniqueName: \"kubernetes.io/projected/c724ef66-462d-429a-8b12-c6b9d7f90a1e-kube-api-access-fwlr4\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.140908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.140754 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.140908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.140777 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.140908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.140808 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.140908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.140832 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.140908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.140864 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh74z\" (UniqueName: \"kubernetes.io/projected/3824e276-5eec-4cea-a06b-c81cc9853ed4-kube-api-access-qh74z\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.140908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.140897 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.141314 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.140925 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.141314 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.140951 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.141314 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.140979 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.141314 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.141257 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.141526 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.141355 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.141526 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.141437 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.141526 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.141488 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.141687 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.141642 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.141744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.141682 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.141800 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.141738 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.141800 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.141780 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.141800 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.141790 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.141964 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.141830 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3824e276-5eec-4cea-a06b-c81cc9853ed4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.141964 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.141886 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.141964 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.141911 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.142146 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.142045 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.142146 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.142106 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.142259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.142171 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.142315 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.142293 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.142646 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.142615 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3824e276-5eec-4cea-a06b-c81cc9853ed4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.143525 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.143504 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.143634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.143572 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.143832 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.143814 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.144366 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.144350 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.158196 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.158142 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.158347 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.158325 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh74z\" (UniqueName: \"kubernetes.io/projected/3824e276-5eec-4cea-a06b-c81cc9853ed4-kube-api-access-qh74z\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.159306 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.159281 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-r9r2z\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.159430 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.159416 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwlr4\" (UniqueName: \"kubernetes.io/projected/c724ef66-462d-429a-8b12-c6b9d7f90a1e-kube-api-access-fwlr4\") pod \"maas-default-gateway-openshift-default-6f54779779-x4r26\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.224206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.224166 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:18.242920 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.242894 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:18.543722 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.543640 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26"] Oct 30 19:19:18.547680 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:19:18.547652 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc724ef66_462d_429a_8b12_c6b9d7f90a1e.slice/crio-187dbf2725895d6aca224201dc248d1089b12662f9c0930d969219d6d518164a WatchSource:0}: Error finding container 187dbf2725895d6aca224201dc248d1089b12662f9c0930d969219d6d518164a: Status 404 returned error can't find the container with id 187dbf2725895d6aca224201dc248d1089b12662f9c0930d969219d6d518164a Oct 30 19:19:18.558533 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:18.558509 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z"] Oct 30 19:19:18.562284 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:19:18.562259 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3824e276_5eec_4cea_a06b_c81cc9853ed4.slice/crio-9b250c5f4d62427c83fd397dff75083d2640add35f8bfbed0b389b04974db894 WatchSource:0}: Error finding container 9b250c5f4d62427c83fd397dff75083d2640add35f8bfbed0b389b04974db894: Status 404 returned error can't find the container with id 9b250c5f4d62427c83fd397dff75083d2640add35f8bfbed0b389b04974db894 Oct 30 19:19:19.105809 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:19.105774 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" event={"ID":"3824e276-5eec-4cea-a06b-c81cc9853ed4","Type":"ContainerStarted","Data":"9b250c5f4d62427c83fd397dff75083d2640add35f8bfbed0b389b04974db894"} Oct 30 19:19:19.106925 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:19.106901 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" event={"ID":"c724ef66-462d-429a-8b12-c6b9d7f90a1e","Type":"ContainerStarted","Data":"187dbf2725895d6aca224201dc248d1089b12662f9c0930d969219d6d518164a"} Oct 30 19:19:21.451504 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:21.451476 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:19:21.451760 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:21.451548 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:19:21.451760 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:21.451579 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:19:21.559350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:21.559315 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:19:21.559445 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:21.559386 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:19:21.559445 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:21.559423 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:19:22.126181 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:22.126141 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" event={"ID":"c724ef66-462d-429a-8b12-c6b9d7f90a1e","Type":"ContainerStarted","Data":"cbbb0054ba2d99fed97e6b3b86d973d24452d560205251334d1460098a2e0ebc"} Oct 30 19:19:22.127632 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:22.127602 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" event={"ID":"3824e276-5eec-4cea-a06b-c81cc9853ed4","Type":"ContainerStarted","Data":"b4face593bc04bd1e82c381924a391418f27208aaff6b94121f0db36eced50a8"} Oct 30 19:19:22.209473 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:22.209422 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" podStartSLOduration=2.30841238 podStartE2EDuration="5.209407694s" totalImagesPullingTime="2.900995314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:22.19952199 +0000 UTC m=+915.273977350" watchObservedRunningTime="2026-10-30 19:19:22.209407694 +0000 UTC m=+915.283863047" Oct 30 19:19:22.225061 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:22.224998 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:22.226549 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:22.226525 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x4r26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 19:19:22.226708 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:22.226568 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" podUID="c724ef66-462d-429a-8b12-c6b9d7f90a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 19:19:22.254030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:22.247595 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:22.254030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:22.249348 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-r9r2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:19:22.254030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:22.249424 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" podUID="3824e276-5eec-4cea-a06b-c81cc9853ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 19:19:22.275029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:22.267725 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" podStartSLOduration=2.273802685 podStartE2EDuration="5.267664971s" totalImagesPullingTime="2.993862286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:22.267099466 +0000 UTC m=+915.341554827" watchObservedRunningTime="2026-10-30 19:19:22.267664971 +0000 UTC m=+915.342120331" Oct 30 19:19:23.225442 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:23.225400 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x4r26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 19:19:23.225850 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:23.225462 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" podUID="c724ef66-462d-429a-8b12-c6b9d7f90a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 19:19:23.258030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:23.256224 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-r9r2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:19:23.258030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:23.256306 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" podUID="3824e276-5eec-4cea-a06b-c81cc9853ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 19:19:24.225498 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:24.225462 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x4r26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 19:19:24.225921 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:24.225520 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" podUID="c724ef66-462d-429a-8b12-c6b9d7f90a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 19:19:24.251031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:24.243893 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-r9r2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:19:24.251031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:24.243966 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" podUID="3824e276-5eec-4cea-a06b-c81cc9853ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 19:19:25.225138 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:25.225096 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x4r26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 19:19:25.225337 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:25.225161 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" podUID="c724ef66-462d-429a-8b12-c6b9d7f90a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 19:19:25.259031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:25.258121 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-r9r2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:19:25.259031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:25.258183 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" podUID="3824e276-5eec-4cea-a06b-c81cc9853ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 19:19:26.225076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:26.225033 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x4r26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 19:19:26.225278 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:26.225088 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" podUID="c724ef66-462d-429a-8b12-c6b9d7f90a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 19:19:26.254043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:26.247984 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-r9r2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:19:26.254043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:26.251186 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" podUID="3824e276-5eec-4cea-a06b-c81cc9853ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 19:19:27.193813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.193787 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5"] Oct 30 19:19:27.198142 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.198120 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" Oct 30 19:19:27.213907 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.213878 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5"] Oct 30 19:19:27.221942 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.221915 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 19:19:27.222429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.222260 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-bmxz7\"" Oct 30 19:19:27.222584 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.222267 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 19:19:27.226960 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.226934 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x4r26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 19:19:27.227214 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.227187 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" podUID="c724ef66-462d-429a-8b12-c6b9d7f90a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 19:19:27.238752 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.238697 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa39929-c809-4d56-ba4a-5b8dcc095774-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-g2nv5\" (UID: \"7fa39929-c809-4d56-ba4a-5b8dcc095774\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" Oct 30 19:19:27.239232 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.239202 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm4v6\" (UniqueName: \"kubernetes.io/projected/7fa39929-c809-4d56-ba4a-5b8dcc095774-kube-api-access-dm4v6\") pod \"ai-gateway-controller-5d9cd99f45-g2nv5\" (UID: \"7fa39929-c809-4d56-ba4a-5b8dcc095774\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" Oct 30 19:19:27.272030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.268365 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-r9r2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:19:27.272351 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.272316 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" podUID="3824e276-5eec-4cea-a06b-c81cc9853ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 19:19:27.389388 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.389353 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa39929-c809-4d56-ba4a-5b8dcc095774-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-g2nv5\" (UID: \"7fa39929-c809-4d56-ba4a-5b8dcc095774\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" Oct 30 19:19:27.389596 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.389472 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm4v6\" (UniqueName: \"kubernetes.io/projected/7fa39929-c809-4d56-ba4a-5b8dcc095774-kube-api-access-dm4v6\") pod \"ai-gateway-controller-5d9cd99f45-g2nv5\" (UID: \"7fa39929-c809-4d56-ba4a-5b8dcc095774\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" Oct 30 19:19:27.392106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.392078 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa39929-c809-4d56-ba4a-5b8dcc095774-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-g2nv5\" (UID: \"7fa39929-c809-4d56-ba4a-5b8dcc095774\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" Oct 30 19:19:27.406451 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.406421 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm4v6\" (UniqueName: \"kubernetes.io/projected/7fa39929-c809-4d56-ba4a-5b8dcc095774-kube-api-access-dm4v6\") pod \"ai-gateway-controller-5d9cd99f45-g2nv5\" (UID: \"7fa39929-c809-4d56-ba4a-5b8dcc095774\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" Oct 30 19:19:27.511602 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.511496 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" Oct 30 19:19:27.837868 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:27.837830 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5"] Oct 30 19:19:27.842128 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:19:27.842092 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa39929_c809_4d56_ba4a_5b8dcc095774.slice/crio-013ee107bab479616ff22403c4b1cd5eba8e8e117d8466ff1bc9d7ee45b6380f WatchSource:0}: Error finding container 013ee107bab479616ff22403c4b1cd5eba8e8e117d8466ff1bc9d7ee45b6380f: Status 404 returned error can't find the container with id 013ee107bab479616ff22403c4b1cd5eba8e8e117d8466ff1bc9d7ee45b6380f Oct 30 19:19:28.202507 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:28.202414 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" event={"ID":"7fa39929-c809-4d56-ba4a-5b8dcc095774","Type":"ContainerStarted","Data":"013ee107bab479616ff22403c4b1cd5eba8e8e117d8466ff1bc9d7ee45b6380f"} Oct 30 19:19:28.225090 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:28.225053 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x4r26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 19:19:28.225283 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:28.225119 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" podUID="c724ef66-462d-429a-8b12-c6b9d7f90a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 19:19:28.243274 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:28.243235 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-r9r2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:19:28.243447 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:28.243291 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" podUID="3824e276-5eec-4cea-a06b-c81cc9853ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 19:19:29.225240 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:29.225197 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-x4r26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 19:19:29.225704 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:29.225269 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" podUID="c724ef66-462d-429a-8b12-c6b9d7f90a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 19:19:29.244183 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:29.244144 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-r9r2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:19:29.244376 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:29.244215 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" podUID="3824e276-5eec-4cea-a06b-c81cc9853ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 19:19:30.530438 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:30.530397 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:30.531279 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:30.531248 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:30.531968 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:30.531937 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:19:30.546039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:30.543650 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:30.546039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:30.544651 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:30.546039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:30.545550 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:19:31.222502 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:31.222454 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" event={"ID":"7fa39929-c809-4d56-ba4a-5b8dcc095774","Type":"ContainerStarted","Data":"ed63ed0f017260ac374f1a2fdd660ce0a6501a0b5f29eef8a4a026f289792087"} Oct 30 19:19:31.222707 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:31.222683 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" Oct 30 19:19:31.256651 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:31.256598 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" podStartSLOduration=1.396605879 podStartE2EDuration="4.256582682s" totalImagesPullingTime="2.859976803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:31.255826579 +0000 UTC m=+924.330281931" watchObservedRunningTime="2026-10-30 19:19:31.256582682 +0000 UTC m=+924.331038041" Oct 30 19:19:33.255929 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.255891 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-mptvz"] Oct 30 19:19:33.259632 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.259606 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-mptvz" Oct 30 19:19:33.283950 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.283923 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-fk2hg\"" Oct 30 19:19:33.286971 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.286941 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-mptvz"] Oct 30 19:19:33.317045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.316986 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 19:19:33.339176 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.339142 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-rspbb"] Oct 30 19:19:33.342776 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.342757 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-rspbb" Oct 30 19:19:33.350832 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.350803 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645e5088-e265-4836-a447-1cb2b21cc050-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-mptvz\" (UID: \"645e5088-e265-4836-a447-1cb2b21cc050\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-mptvz" Oct 30 19:19:33.350997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.350970 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvdrq\" (UniqueName: \"kubernetes.io/projected/645e5088-e265-4836-a447-1cb2b21cc050-kube-api-access-mvdrq\") pod \"payload-pre-processing-7b98bb678c-mptvz\" (UID: \"645e5088-e265-4836-a447-1cb2b21cc050\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-mptvz" Oct 30 19:19:33.361107 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.361075 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-rspbb"] Oct 30 19:19:33.452769 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.452723 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645e5088-e265-4836-a447-1cb2b21cc050-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-mptvz\" (UID: \"645e5088-e265-4836-a447-1cb2b21cc050\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-mptvz" Oct 30 19:19:33.452948 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.452836 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a-plugins-config-volume\") pod \"payload-processing-5f78687d5b-rspbb\" (UID: \"8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a\") " pod="openshift-ingress/payload-processing-5f78687d5b-rspbb" Oct 30 19:19:33.452948 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.452890 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvdrq\" (UniqueName: \"kubernetes.io/projected/645e5088-e265-4836-a447-1cb2b21cc050-kube-api-access-mvdrq\") pod \"payload-pre-processing-7b98bb678c-mptvz\" (UID: \"645e5088-e265-4836-a447-1cb2b21cc050\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-mptvz" Oct 30 19:19:33.452948 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.452916 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7d8j\" (UniqueName: \"kubernetes.io/projected/8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a-kube-api-access-s7d8j\") pod \"payload-processing-5f78687d5b-rspbb\" (UID: \"8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a\") " pod="openshift-ingress/payload-processing-5f78687d5b-rspbb" Oct 30 19:19:33.453568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.453548 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645e5088-e265-4836-a447-1cb2b21cc050-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-mptvz\" (UID: \"645e5088-e265-4836-a447-1cb2b21cc050\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-mptvz" Oct 30 19:19:33.470095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.470068 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvdrq\" (UniqueName: \"kubernetes.io/projected/645e5088-e265-4836-a447-1cb2b21cc050-kube-api-access-mvdrq\") pod \"payload-pre-processing-7b98bb678c-mptvz\" (UID: \"645e5088-e265-4836-a447-1cb2b21cc050\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-mptvz" Oct 30 19:19:33.553675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.553587 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a-plugins-config-volume\") pod \"payload-processing-5f78687d5b-rspbb\" (UID: \"8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a\") " pod="openshift-ingress/payload-processing-5f78687d5b-rspbb" Oct 30 19:19:33.553675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.553640 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7d8j\" (UniqueName: \"kubernetes.io/projected/8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a-kube-api-access-s7d8j\") pod \"payload-processing-5f78687d5b-rspbb\" (UID: \"8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a\") " pod="openshift-ingress/payload-processing-5f78687d5b-rspbb" Oct 30 19:19:33.554350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.554328 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a-plugins-config-volume\") pod \"payload-processing-5f78687d5b-rspbb\" (UID: \"8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a\") " pod="openshift-ingress/payload-processing-5f78687d5b-rspbb" Oct 30 19:19:33.573391 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.573365 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-mptvz" Oct 30 19:19:33.573940 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.573914 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7d8j\" (UniqueName: \"kubernetes.io/projected/8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a-kube-api-access-s7d8j\") pod \"payload-processing-5f78687d5b-rspbb\" (UID: \"8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a\") " pod="openshift-ingress/payload-processing-5f78687d5b-rspbb" Oct 30 19:19:33.657044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.656726 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-rspbb" Oct 30 19:19:33.886607 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.886571 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-mptvz"] Oct 30 19:19:33.892934 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:19:33.892900 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645e5088_e265_4836_a447_1cb2b21cc050.slice/crio-53548574c42f40073ae0cebd83179b3002b1a170dba0a9dbd8eef6aeb5509a48 WatchSource:0}: Error finding container 53548574c42f40073ae0cebd83179b3002b1a170dba0a9dbd8eef6aeb5509a48: Status 404 returned error can't find the container with id 53548574c42f40073ae0cebd83179b3002b1a170dba0a9dbd8eef6aeb5509a48 Oct 30 19:19:33.975020 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:33.974970 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-rspbb"] Oct 30 19:19:33.979224 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:19:33.979197 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa7fdd5_24e6_4482_9bcc_91fe3c45b92a.slice/crio-3d120b77f720370167f25694165142c7869563869e04870804e17123c7741ea4 WatchSource:0}: Error finding container 3d120b77f720370167f25694165142c7869563869e04870804e17123c7741ea4: Status 404 returned error can't find the container with id 3d120b77f720370167f25694165142c7869563869e04870804e17123c7741ea4 Oct 30 19:19:34.244717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:34.244620 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-mptvz" event={"ID":"645e5088-e265-4836-a447-1cb2b21cc050","Type":"ContainerStarted","Data":"53548574c42f40073ae0cebd83179b3002b1a170dba0a9dbd8eef6aeb5509a48"} Oct 30 19:19:34.246119 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:34.246093 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-rspbb" event={"ID":"8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a","Type":"ContainerStarted","Data":"3d120b77f720370167f25694165142c7869563869e04870804e17123c7741ea4"} Oct 30 19:19:37.272228 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:37.272186 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-rspbb" event={"ID":"8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a","Type":"ContainerStarted","Data":"4a58bfa50d53753cd6074ed39035c8b8cec14ca50a0365bc3564f10dfdef25b2"} Oct 30 19:19:37.272713 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:37.272256 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-rspbb" Oct 30 19:19:37.273745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:37.273716 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-mptvz" event={"ID":"645e5088-e265-4836-a447-1cb2b21cc050","Type":"ContainerStarted","Data":"d9939db7ea24b4cb6b71831f1c9aa8591cfcc400af70bba56e7fad3a98675e39"} Oct 30 19:19:37.273850 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:37.273835 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-mptvz" Oct 30 19:19:37.305150 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:37.304904 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-rspbb" podStartSLOduration=1.231877657 podStartE2EDuration="4.304889205s" totalImagesPullingTime="3.073011548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:37.304713907 +0000 UTC m=+930.379169266" watchObservedRunningTime="2026-10-30 19:19:37.304889205 +0000 UTC m=+930.379344560" Oct 30 19:19:37.339794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:37.339740 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-mptvz" podStartSLOduration=1.18215572 podStartE2EDuration="4.339726942s" totalImagesPullingTime="3.157571222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:37.337493014 +0000 UTC m=+930.411948373" watchObservedRunningTime="2026-10-30 19:19:37.339726942 +0000 UTC m=+930.414182301" Oct 30 19:19:39.599909 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:19:39.599868 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:19:42.229344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:42.229308 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" Oct 30 19:19:42.506109 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:42.506018 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-84dd667bd6-hz7tl"] Oct 30 19:19:42.511159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:42.511132 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dd667bd6-hz7tl" Oct 30 19:19:42.527098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:42.527067 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-84dd667bd6-hz7tl"] Oct 30 19:19:42.563400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:42.563364 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3977e82d-a7b6-4f25-aa74-8d1f6488ff41-metrics-certs\") pod \"ai-gateway-controller-84dd667bd6-hz7tl\" (UID: \"3977e82d-a7b6-4f25-aa74-8d1f6488ff41\") " pod="opendatahub/ai-gateway-controller-84dd667bd6-hz7tl" Oct 30 19:19:42.563578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:42.563405 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jc74\" (UniqueName: \"kubernetes.io/projected/3977e82d-a7b6-4f25-aa74-8d1f6488ff41-kube-api-access-4jc74\") pod \"ai-gateway-controller-84dd667bd6-hz7tl\" (UID: \"3977e82d-a7b6-4f25-aa74-8d1f6488ff41\") " pod="opendatahub/ai-gateway-controller-84dd667bd6-hz7tl" Oct 30 19:19:42.664769 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:42.664733 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3977e82d-a7b6-4f25-aa74-8d1f6488ff41-metrics-certs\") pod \"ai-gateway-controller-84dd667bd6-hz7tl\" (UID: \"3977e82d-a7b6-4f25-aa74-8d1f6488ff41\") " pod="opendatahub/ai-gateway-controller-84dd667bd6-hz7tl" Oct 30 19:19:42.664769 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:42.664776 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jc74\" (UniqueName: \"kubernetes.io/projected/3977e82d-a7b6-4f25-aa74-8d1f6488ff41-kube-api-access-4jc74\") pod \"ai-gateway-controller-84dd667bd6-hz7tl\" (UID: \"3977e82d-a7b6-4f25-aa74-8d1f6488ff41\") " pod="opendatahub/ai-gateway-controller-84dd667bd6-hz7tl" Oct 30 19:19:42.667245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:42.667215 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3977e82d-a7b6-4f25-aa74-8d1f6488ff41-metrics-certs\") pod \"ai-gateway-controller-84dd667bd6-hz7tl\" (UID: \"3977e82d-a7b6-4f25-aa74-8d1f6488ff41\") " pod="opendatahub/ai-gateway-controller-84dd667bd6-hz7tl" Oct 30 19:19:42.682545 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:42.682515 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jc74\" (UniqueName: \"kubernetes.io/projected/3977e82d-a7b6-4f25-aa74-8d1f6488ff41-kube-api-access-4jc74\") pod \"ai-gateway-controller-84dd667bd6-hz7tl\" (UID: \"3977e82d-a7b6-4f25-aa74-8d1f6488ff41\") " pod="opendatahub/ai-gateway-controller-84dd667bd6-hz7tl" Oct 30 19:19:42.824159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:42.824059 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dd667bd6-hz7tl" Oct 30 19:19:43.117680 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:43.117639 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-84dd667bd6-hz7tl"] Oct 30 19:19:43.120615 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:19:43.120589 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3977e82d_a7b6_4f25_aa74_8d1f6488ff41.slice/crio-7d1a8355af79cc1de029d94b7b6899fde90d57b41cf133f261fb3ba9fef052f8 WatchSource:0}: Error finding container 7d1a8355af79cc1de029d94b7b6899fde90d57b41cf133f261fb3ba9fef052f8: Status 404 returned error can't find the container with id 7d1a8355af79cc1de029d94b7b6899fde90d57b41cf133f261fb3ba9fef052f8 Oct 30 19:19:43.302714 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:43.302673 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dd667bd6-hz7tl" event={"ID":"3977e82d-a7b6-4f25-aa74-8d1f6488ff41","Type":"ContainerStarted","Data":"7d1a8355af79cc1de029d94b7b6899fde90d57b41cf133f261fb3ba9fef052f8"} Oct 30 19:19:44.308983 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:44.308947 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dd667bd6-hz7tl" event={"ID":"3977e82d-a7b6-4f25-aa74-8d1f6488ff41","Type":"ContainerStarted","Data":"c6e9b458c0c7169ad879df8e98359398dd8e240e0ea8bc8a063df4afd90aaaf2"} Oct 30 19:19:44.309481 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:44.309026 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-84dd667bd6-hz7tl" Oct 30 19:19:44.345472 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:44.345410 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-84dd667bd6-hz7tl" podStartSLOduration=1.975673487 podStartE2EDuration="2.345394126s" totalImagesPullingTime="369.720639ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:44.343461419 +0000 UTC m=+937.417916777" watchObservedRunningTime="2026-10-30 19:19:44.345394126 +0000 UTC m=+937.419849513" Oct 30 19:19:48.281819 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:48.281738 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-mptvz" Oct 30 19:19:48.282288 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:48.281837 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-rspbb" Oct 30 19:19:55.315693 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:55.315653 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-84dd667bd6-hz7tl" Oct 30 19:19:55.380930 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:55.380893 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5"] Oct 30 19:19:55.381224 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:55.381188 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" podUID="7fa39929-c809-4d56-ba4a-5b8dcc095774" containerName="manager" containerID="cri-o://ed63ed0f017260ac374f1a2fdd660ce0a6501a0b5f29eef8a4a026f289792087" gracePeriod=30 Oct 30 19:19:55.645677 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:55.645650 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" Oct 30 19:19:55.690819 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:55.690780 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa39929-c809-4d56-ba4a-5b8dcc095774-kube-api-access-dm4v6\" (UniqueName: \"kubernetes.io/projected/7fa39929-c809-4d56-ba4a-5b8dcc095774-kube-api-access-dm4v6\") pod \"7fa39929-c809-4d56-ba4a-5b8dcc095774\" (UID: \"7fa39929-c809-4d56-ba4a-5b8dcc095774\") " Oct 30 19:19:55.691026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:55.690928 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa39929-c809-4d56-ba4a-5b8dcc095774-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa39929-c809-4d56-ba4a-5b8dcc095774-metrics-certs\") pod \"7fa39929-c809-4d56-ba4a-5b8dcc095774\" (UID: \"7fa39929-c809-4d56-ba4a-5b8dcc095774\") " Oct 30 19:19:55.693506 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:55.693472 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa39929-c809-4d56-ba4a-5b8dcc095774-kube-api-access-dm4v6" pod "7fa39929-c809-4d56-ba4a-5b8dcc095774" (UID: "7fa39929-c809-4d56-ba4a-5b8dcc095774"). InnerVolumeSpecName "kube-api-access-dm4v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:19:55.693680 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:55.693508 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa39929-c809-4d56-ba4a-5b8dcc095774-metrics-certs" pod "7fa39929-c809-4d56-ba4a-5b8dcc095774" (UID: "7fa39929-c809-4d56-ba4a-5b8dcc095774"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:19:55.792083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:55.792042 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa39929-c809-4d56-ba4a-5b8dcc095774-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:19:55.792331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:55.792143 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm4v6\" (UniqueName: \"kubernetes.io/projected/7fa39929-c809-4d56-ba4a-5b8dcc095774-kube-api-access-dm4v6\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:19:56.368951 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:56.368910 5047 generic.go:356] "Generic (PLEG): container finished" podID="7fa39929-c809-4d56-ba4a-5b8dcc095774" containerID="ed63ed0f017260ac374f1a2fdd660ce0a6501a0b5f29eef8a4a026f289792087" exitCode=0 Oct 30 19:19:56.369404 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:56.368959 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" event={"ID":"7fa39929-c809-4d56-ba4a-5b8dcc095774","Type":"ContainerDied","Data":"ed63ed0f017260ac374f1a2fdd660ce0a6501a0b5f29eef8a4a026f289792087"} Oct 30 19:19:56.369404 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:56.368999 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" event={"ID":"7fa39929-c809-4d56-ba4a-5b8dcc095774","Type":"ContainerDied","Data":"013ee107bab479616ff22403c4b1cd5eba8e8e117d8466ff1bc9d7ee45b6380f"} Oct 30 19:19:56.369404 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:56.368963 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5" Oct 30 19:19:56.369404 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:56.369047 5047 scope.go:122] "RemoveContainer" containerID="ed63ed0f017260ac374f1a2fdd660ce0a6501a0b5f29eef8a4a026f289792087" Oct 30 19:19:56.378664 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:56.378642 5047 scope.go:122] "RemoveContainer" containerID="ed63ed0f017260ac374f1a2fdd660ce0a6501a0b5f29eef8a4a026f289792087" Oct 30 19:19:56.378937 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:19:56.378919 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed63ed0f017260ac374f1a2fdd660ce0a6501a0b5f29eef8a4a026f289792087\": container with ID starting with ed63ed0f017260ac374f1a2fdd660ce0a6501a0b5f29eef8a4a026f289792087 not found: ID does not exist" containerID="ed63ed0f017260ac374f1a2fdd660ce0a6501a0b5f29eef8a4a026f289792087" Oct 30 19:19:56.379035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:56.378943 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed63ed0f017260ac374f1a2fdd660ce0a6501a0b5f29eef8a4a026f289792087"} err="failed to get container status \"ed63ed0f017260ac374f1a2fdd660ce0a6501a0b5f29eef8a4a026f289792087\": rpc error: code = NotFound desc = could not find container \"ed63ed0f017260ac374f1a2fdd660ce0a6501a0b5f29eef8a4a026f289792087\": container with ID starting with ed63ed0f017260ac374f1a2fdd660ce0a6501a0b5f29eef8a4a026f289792087 not found: ID does not exist" Oct 30 19:19:56.419066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:56.419024 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5"] Oct 30 19:19:56.430535 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:56.430503 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-g2nv5"] Oct 30 19:19:57.047515 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:19:57.047482 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fa39929-c809-4d56-ba4a-5b8dcc095774" path="/var/lib/kubelet/pods/7fa39929-c809-4d56-ba4a-5b8dcc095774/volumes" Oct 30 19:20:09.601117 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:20:09.601076 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:20:39.599615 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:20:39.599578 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:21:09.600868 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:21:09.600825 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:21:14.572316 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.572272 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd"] Oct 30 19:21:14.573045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.573022 5047 cpu_manager.go:414] "containerMap: removing container" podUID="7fa39929-c809-4d56-ba4a-5b8dcc095774" containerName="manager" Oct 30 19:21:14.573111 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.573049 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa39929-c809-4d56-ba4a-5b8dcc095774" containerName="manager" Oct 30 19:21:14.573233 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.573220 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa39929-c809-4d56-ba4a-5b8dcc095774" containerName="manager" Oct 30 19:21:14.576858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.576836 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.600345 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.600309 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd"] Oct 30 19:21:14.609276 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.609240 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9efc72-openshift-default-dockercfg-jrkpb\"" Oct 30 19:21:14.629174 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.629135 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.629174 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.629183 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.629576 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.629555 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.629673 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.629644 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zftl\" (UniqueName: \"kubernetes.io/projected/9e15d741-48fd-4962-90db-de0e6cb664f3-kube-api-access-8zftl\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.629772 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.629757 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.629831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.629797 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.629890 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.629871 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.629934 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.629907 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e15d741-48fd-4962-90db-de0e6cb664f3-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.629934 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.629926 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.730924 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.730878 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.730924 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.730925 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e15d741-48fd-4962-90db-de0e6cb664f3-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.731224 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.730945 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.731224 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.731112 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.731224 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.731163 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.731419 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.731398 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.731505 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.731476 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.731564 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.731529 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.731625 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.731566 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zftl\" (UniqueName: \"kubernetes.io/projected/9e15d741-48fd-4962-90db-de0e6cb664f3-kube-api-access-8zftl\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.731686 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.731622 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.731686 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.731662 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.731822 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.731794 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e15d741-48fd-4962-90db-de0e6cb664f3-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.731879 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.731801 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.731992 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.731971 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.734283 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.734251 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.744663 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.744608 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.749994 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.749965 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.750471 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.750437 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zftl\" (UniqueName: \"kubernetes.io/projected/9e15d741-48fd-4962-90db-de0e6cb664f3-kube-api-access-8zftl\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.823521 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.823438 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt"] Oct 30 19:21:14.829561 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.828829 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:14.853835 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.853801 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt"] Oct 30 19:21:14.891647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.891609 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:14.912635 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.912595 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-249dd4-openshift-default-dockercfg-gbt9k\"" Oct 30 19:21:14.934611 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.934385 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-token\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:14.934611 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.934445 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54lrn\" (UniqueName: \"kubernetes.io/projected/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-kube-api-access-54lrn\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:14.934611 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.934484 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-workload-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:14.934611 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.934517 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-podinfo\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:14.934611 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.934560 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-data\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:14.934611 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.934602 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istiod-ca-cert\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:14.935069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.934687 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-envoy\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:14.935069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.934904 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-credential-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:14.935069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:14.934996 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-workload-certs\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.036120 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.036085 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-workload-certs\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.036302 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.036137 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-token\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.036302 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.036169 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54lrn\" (UniqueName: \"kubernetes.io/projected/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-kube-api-access-54lrn\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.036302 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.036195 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-workload-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.036302 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.036221 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-podinfo\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.036302 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.036259 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-data\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.036302 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.036298 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istiod-ca-cert\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.036503 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.036333 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-envoy\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.036503 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.036403 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-credential-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.036592 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.036563 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-workload-certs\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.036834 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.036808 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-credential-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.037109 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.037032 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-workload-socket\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.037109 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.037104 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-data\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.037424 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.037403 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istiod-ca-cert\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.039858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.039547 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-podinfo\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.040309 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.040286 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-envoy\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.055479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.055449 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-token\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.055623 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.055581 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54lrn\" (UniqueName: \"kubernetes.io/projected/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-kube-api-access-54lrn\") pod \"e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.102178 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.101662 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j"] Oct 30 19:21:15.107509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.107479 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.126333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.126129 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5ac614-openshift-default-dockercfg-2npc8\"" Oct 30 19:21:15.129148 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.129118 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j"] Oct 30 19:21:15.137176 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.137140 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.137343 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.137312 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.137446 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.137418 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.137571 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.137460 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.137571 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.137504 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvshj\" (UniqueName: \"kubernetes.io/projected/9aff363b-dce0-48d0-91cc-7783972b1f85-kube-api-access-tvshj\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.137571 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.137542 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.137741 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.137571 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aff363b-dce0-48d0-91cc-7783972b1f85-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.137799 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.137766 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.137852 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.137811 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.148682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.148653 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:15.211922 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:21:15.211892 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e15d741_48fd_4962_90db_de0e6cb664f3.slice/crio-5ec5ecdafcd62554dee89f418e7ae4d08b1755a075eb48b8b603c3e1d47cbcad WatchSource:0}: Error finding container 5ec5ecdafcd62554dee89f418e7ae4d08b1755a075eb48b8b603c3e1d47cbcad: Status 404 returned error can't find the container with id 5ec5ecdafcd62554dee89f418e7ae4d08b1755a075eb48b8b603c3e1d47cbcad Oct 30 19:21:15.212279 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.212256 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd"] Oct 30 19:21:15.216053 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.215300 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:15.216053 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.215390 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:15.216053 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.215434 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:15.239303 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.239267 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.239432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.239330 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.239432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.239402 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.239548 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.239473 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.239548 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.239504 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.239548 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.239536 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.239656 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.239578 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvshj\" (UniqueName: \"kubernetes.io/projected/9aff363b-dce0-48d0-91cc-7783972b1f85-kube-api-access-tvshj\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.239656 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.239617 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.239656 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.239645 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aff363b-dce0-48d0-91cc-7783972b1f85-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.240503 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.240469 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aff363b-dce0-48d0-91cc-7783972b1f85-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.241778 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.240747 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.242084 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.240986 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.242084 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.241208 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.242084 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.241230 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.242716 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.242694 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.245508 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.245484 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.259876 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.259811 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.260069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.260046 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvshj\" (UniqueName: \"kubernetes.io/projected/9aff363b-dce0-48d0-91cc-7783972b1f85-kube-api-access-tvshj\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.432958 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.432868 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:15.460637 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.460601 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt"] Oct 30 19:21:15.462097 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:21:15.462063 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd1d5de_7f02_4d79_a762_ee3d77c86bba.slice/crio-9a7154ec79213135dd3570487ec91a2cda3b2f5980de9748183954f23a11fa64 WatchSource:0}: Error finding container 9a7154ec79213135dd3570487ec91a2cda3b2f5980de9748183954f23a11fa64: Status 404 returned error can't find the container with id 9a7154ec79213135dd3570487ec91a2cda3b2f5980de9748183954f23a11fa64 Oct 30 19:21:15.464911 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.464869 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:15.464992 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.464971 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:15.465054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.465036 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:15.745325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.745176 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j"] Oct 30 19:21:15.747792 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:21:15.747763 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aff363b_dce0_48d0_91cc_7783972b1f85.slice/crio-b909da1c97c987498a178541efb4983cb0ad1f0611e818521265cf29479da374 WatchSource:0}: Error finding container b909da1c97c987498a178541efb4983cb0ad1f0611e818521265cf29479da374: Status 404 returned error can't find the container with id b909da1c97c987498a178541efb4983cb0ad1f0611e818521265cf29479da374 Oct 30 19:21:15.750371 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.750332 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:15.750486 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.750403 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:15.750486 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.750432 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:15.765788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.765687 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" event={"ID":"8fd1d5de-7f02-4d79-a762-ee3d77c86bba","Type":"ContainerStarted","Data":"56763c4475020b081e4a70044ea983e9e32616c596850747719576485bb8d11c"} Oct 30 19:21:15.765788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.765751 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" event={"ID":"8fd1d5de-7f02-4d79-a762-ee3d77c86bba","Type":"ContainerStarted","Data":"9a7154ec79213135dd3570487ec91a2cda3b2f5980de9748183954f23a11fa64"} Oct 30 19:21:15.768057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.767997 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" event={"ID":"9e15d741-48fd-4962-90db-de0e6cb664f3","Type":"ContainerStarted","Data":"7b4274f5e71fc6a82355b76c97e5554c897e19ce89d3a0c939945db10ff1e3e2"} Oct 30 19:21:15.768175 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.768086 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" event={"ID":"9e15d741-48fd-4962-90db-de0e6cb664f3","Type":"ContainerStarted","Data":"5ec5ecdafcd62554dee89f418e7ae4d08b1755a075eb48b8b603c3e1d47cbcad"} Oct 30 19:21:15.769921 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.769893 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" event={"ID":"9aff363b-dce0-48d0-91cc-7783972b1f85","Type":"ContainerStarted","Data":"b909da1c97c987498a178541efb4983cb0ad1f0611e818521265cf29479da374"} Oct 30 19:21:15.806090 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.805995 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" podStartSLOduration=1.805980205 podStartE2EDuration="1.805980205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:15.804935611 +0000 UTC m=+1028.879390975" watchObservedRunningTime="2026-10-30 19:21:15.805980205 +0000 UTC m=+1028.880435634" Oct 30 19:21:15.846989 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.846924 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" podStartSLOduration=1.846901055 podStartE2EDuration="1.846901055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:15.844357332 +0000 UTC m=+1028.918812691" watchObservedRunningTime="2026-10-30 19:21:15.846901055 +0000 UTC m=+1028.921356414" Oct 30 19:21:15.892418 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:15.892386 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:16.149001 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:16.148969 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:16.154457 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:16.154426 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:16.776736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:16.776694 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" event={"ID":"9aff363b-dce0-48d0-91cc-7783972b1f85","Type":"ContainerStarted","Data":"919f6e2b4a4060e86af745524d5294dd59dabb80f1caf4d3d772f1155fb2a80f"} Oct 30 19:21:16.777192 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:16.777121 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:16.777953 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:16.777930 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:21:16.817053 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:16.816978 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" podStartSLOduration=1.816962427 podStartE2EDuration="1.816962427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:16.815866567 +0000 UTC m=+1029.890321926" watchObservedRunningTime="2026-10-30 19:21:16.816962427 +0000 UTC m=+1029.891417785" Oct 30 19:21:16.897540 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:16.897497 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:17.433106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:17.433070 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:17.439053 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:17.439024 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:17.784463 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:17.784371 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:17.784463 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:17.784429 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:17.784996 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:17.784978 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:17.785067 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:17.785055 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:20.338265 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:20.338224 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr"] Oct 30 19:21:20.343159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:20.343134 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" Oct 30 19:21:20.363868 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:20.363821 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-9efc72-dockercfg-clwft\"" Oct 30 19:21:20.377614 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:20.377579 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-9efc72\"" Oct 30 19:21:20.386021 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:20.385966 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr"] Oct 30 19:21:20.397595 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:20.397562 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ff13b0-8eb3-474b-8579-29b192ccd9ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr\" (UID: \"71ff13b0-8eb3-474b-8579-29b192ccd9ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" Oct 30 19:21:20.397767 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:20.397613 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88r8b\" (UniqueName: \"kubernetes.io/projected/71ff13b0-8eb3-474b-8579-29b192ccd9ee-kube-api-access-88r8b\") pod \"payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr\" (UID: \"71ff13b0-8eb3-474b-8579-29b192ccd9ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" Oct 30 19:21:20.499154 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:20.499118 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ff13b0-8eb3-474b-8579-29b192ccd9ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr\" (UID: \"71ff13b0-8eb3-474b-8579-29b192ccd9ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" Oct 30 19:21:20.499154 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:20.499158 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88r8b\" (UniqueName: \"kubernetes.io/projected/71ff13b0-8eb3-474b-8579-29b192ccd9ee-kube-api-access-88r8b\") pod \"payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr\" (UID: \"71ff13b0-8eb3-474b-8579-29b192ccd9ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" Oct 30 19:21:20.499882 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:20.499855 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ff13b0-8eb3-474b-8579-29b192ccd9ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr\" (UID: \"71ff13b0-8eb3-474b-8579-29b192ccd9ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" Oct 30 19:21:20.519088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:20.519056 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88r8b\" (UniqueName: \"kubernetes.io/projected/71ff13b0-8eb3-474b-8579-29b192ccd9ee-kube-api-access-88r8b\") pod \"payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr\" (UID: \"71ff13b0-8eb3-474b-8579-29b192ccd9ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" Oct 30 19:21:20.665354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:20.665259 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" Oct 30 19:21:20.962536 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:20.962507 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr"] Oct 30 19:21:20.964383 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:21:20.964353 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ff13b0_8eb3_474b_8579_29b192ccd9ee.slice/crio-97e4a938b07d321927645efc4bfdc110ff7c7f9ae0761670130530f2ce5a0c57 WatchSource:0}: Error finding container 97e4a938b07d321927645efc4bfdc110ff7c7f9ae0761670130530f2ce5a0c57: Status 404 returned error can't find the container with id 97e4a938b07d321927645efc4bfdc110ff7c7f9ae0761670130530f2ce5a0c57 Oct 30 19:21:21.429531 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.429490 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995"] Oct 30 19:21:21.433747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.433726 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 19:21:21.452960 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.452924 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995"] Oct 30 19:21:21.493752 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.493719 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-9efc72-serving-cert\"" Oct 30 19:21:21.508652 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.508617 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12a3bdc-6b65-4496-8458-ff694a332732-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9efc72-59d769f774-6s995\" (UID: \"a12a3bdc-6b65-4496-8458-ff694a332732\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 19:21:21.508845 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.508790 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm6bc\" (UniqueName: \"kubernetes.io/projected/a12a3bdc-6b65-4496-8458-ff694a332732-kube-api-access-vm6bc\") pod \"maas-api-e2e-worker-w2-9efc72-59d769f774-6s995\" (UID: \"a12a3bdc-6b65-4496-8458-ff694a332732\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 19:21:21.508845 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.508827 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12a3bdc-6b65-4496-8458-ff694a332732-metrics-certs\") pod \"maas-api-e2e-worker-w2-9efc72-59d769f774-6s995\" (UID: \"a12a3bdc-6b65-4496-8458-ff694a332732\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 19:21:21.508971 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.508894 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12a3bdc-6b65-4496-8458-ff694a332732-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9efc72-59d769f774-6s995\" (UID: \"a12a3bdc-6b65-4496-8458-ff694a332732\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 19:21:21.619817 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.619774 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm6bc\" (UniqueName: \"kubernetes.io/projected/a12a3bdc-6b65-4496-8458-ff694a332732-kube-api-access-vm6bc\") pod \"maas-api-e2e-worker-w2-9efc72-59d769f774-6s995\" (UID: \"a12a3bdc-6b65-4496-8458-ff694a332732\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 19:21:21.620108 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.620087 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12a3bdc-6b65-4496-8458-ff694a332732-metrics-certs\") pod \"maas-api-e2e-worker-w2-9efc72-59d769f774-6s995\" (UID: \"a12a3bdc-6b65-4496-8458-ff694a332732\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 19:21:21.620268 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.620252 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12a3bdc-6b65-4496-8458-ff694a332732-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9efc72-59d769f774-6s995\" (UID: \"a12a3bdc-6b65-4496-8458-ff694a332732\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 19:21:21.620471 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.620455 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12a3bdc-6b65-4496-8458-ff694a332732-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9efc72-59d769f774-6s995\" (UID: \"a12a3bdc-6b65-4496-8458-ff694a332732\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 19:21:21.624721 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.621438 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12a3bdc-6b65-4496-8458-ff694a332732-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9efc72-59d769f774-6s995\" (UID: \"a12a3bdc-6b65-4496-8458-ff694a332732\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 19:21:21.624986 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:21:21.624965 5047 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-serving-cert: secret "maas-api-e2e-worker-w2-9efc72-serving-cert" not found Oct 30 19:21:21.625166 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:21:21.625150 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a12a3bdc-6b65-4496-8458-ff694a332732-maas-api-tls podName:a12a3bdc-6b65-4496-8458-ff694a332732 nodeName:}" failed. No retries permitted until 2026-10-30 19:21:22.125131736 +0000 UTC m=+1035.199587085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a12a3bdc-6b65-4496-8458-ff694a332732-maas-api-tls") pod "maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" (UID: "a12a3bdc-6b65-4496-8458-ff694a332732") : secret "maas-api-e2e-worker-w2-9efc72-serving-cert" not found Oct 30 19:21:21.630933 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.630901 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12a3bdc-6b65-4496-8458-ff694a332732-metrics-certs\") pod \"maas-api-e2e-worker-w2-9efc72-59d769f774-6s995\" (UID: \"a12a3bdc-6b65-4496-8458-ff694a332732\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 19:21:21.640578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.640546 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm6bc\" (UniqueName: \"kubernetes.io/projected/a12a3bdc-6b65-4496-8458-ff694a332732-kube-api-access-vm6bc\") pod \"maas-api-e2e-worker-w2-9efc72-59d769f774-6s995\" (UID: \"a12a3bdc-6b65-4496-8458-ff694a332732\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 19:21:21.805920 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.805822 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" event={"ID":"71ff13b0-8eb3-474b-8579-29b192ccd9ee","Type":"ContainerStarted","Data":"f7934e0fb31562f7d0f762abdc8c3ebb373602d38c9473c132056d0e9ea71180"} Oct 30 19:21:21.805920 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.805882 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" event={"ID":"71ff13b0-8eb3-474b-8579-29b192ccd9ee","Type":"ContainerStarted","Data":"97e4a938b07d321927645efc4bfdc110ff7c7f9ae0761670130530f2ce5a0c57"} Oct 30 19:21:21.805920 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.805910 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" Oct 30 19:21:21.843964 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:21.843906 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" podStartSLOduration=1.843893762 podStartE2EDuration="1.843893762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:21.841734756 +0000 UTC m=+1034.916190115" watchObservedRunningTime="2026-10-30 19:21:21.843893762 +0000 UTC m=+1034.918349120" Oct 30 19:21:22.126404 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:22.126365 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12a3bdc-6b65-4496-8458-ff694a332732-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9efc72-59d769f774-6s995\" (UID: \"a12a3bdc-6b65-4496-8458-ff694a332732\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 19:21:22.128940 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:22.128909 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12a3bdc-6b65-4496-8458-ff694a332732-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9efc72-59d769f774-6s995\" (UID: \"a12a3bdc-6b65-4496-8458-ff694a332732\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 19:21:22.346685 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:22.346648 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 19:21:22.661075 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:22.661047 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995"] Oct 30 19:21:22.663047 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:21:22.663016 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12a3bdc_6b65_4496_8458_ff694a332732.slice/crio-3345e2304d7a1c43358a14c84dad6a79f1c83343aceacebd715c8b59ee3f7f4c WatchSource:0}: Error finding container 3345e2304d7a1c43358a14c84dad6a79f1c83343aceacebd715c8b59ee3f7f4c: Status 404 returned error can't find the container with id 3345e2304d7a1c43358a14c84dad6a79f1c83343aceacebd715c8b59ee3f7f4c Oct 30 19:21:22.664723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:22.664701 5047 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:21:22.813372 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:22.813332 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" event={"ID":"a12a3bdc-6b65-4496-8458-ff694a332732","Type":"ContainerStarted","Data":"3345e2304d7a1c43358a14c84dad6a79f1c83343aceacebd715c8b59ee3f7f4c"} Oct 30 19:21:23.123943 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:23.120751 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2"] Oct 30 19:21:23.128462 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:23.128180 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" Oct 30 19:21:23.143670 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:23.143591 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2"] Oct 30 19:21:23.152490 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:23.152454 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5ac614-dockercfg-7f9qz\"" Oct 30 19:21:23.152490 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:23.152454 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5ac614\"" Oct 30 19:21:23.242689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:23.242654 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54643a49-f1da-4957-a6d6-81fb67991b2e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2\" (UID: \"54643a49-f1da-4957-a6d6-81fb67991b2e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" Oct 30 19:21:23.242829 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:23.242758 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl8cw\" (UniqueName: \"kubernetes.io/projected/54643a49-f1da-4957-a6d6-81fb67991b2e-kube-api-access-fl8cw\") pod \"payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2\" (UID: \"54643a49-f1da-4957-a6d6-81fb67991b2e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" Oct 30 19:21:23.343675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:23.343630 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54643a49-f1da-4957-a6d6-81fb67991b2e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2\" (UID: \"54643a49-f1da-4957-a6d6-81fb67991b2e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" Oct 30 19:21:23.343876 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:23.343700 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl8cw\" (UniqueName: \"kubernetes.io/projected/54643a49-f1da-4957-a6d6-81fb67991b2e-kube-api-access-fl8cw\") pod \"payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2\" (UID: \"54643a49-f1da-4957-a6d6-81fb67991b2e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" Oct 30 19:21:23.344389 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:23.344367 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54643a49-f1da-4957-a6d6-81fb67991b2e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2\" (UID: \"54643a49-f1da-4957-a6d6-81fb67991b2e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" Oct 30 19:21:23.362183 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:23.362156 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl8cw\" (UniqueName: \"kubernetes.io/projected/54643a49-f1da-4957-a6d6-81fb67991b2e-kube-api-access-fl8cw\") pod \"payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2\" (UID: \"54643a49-f1da-4957-a6d6-81fb67991b2e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" Oct 30 19:21:23.468968 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:23.468935 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" Oct 30 19:21:23.805276 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:23.802431 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2"] Oct 30 19:21:23.805626 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:21:23.805593 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54643a49_f1da_4957_a6d6_81fb67991b2e.slice/crio-24446ee291af9dd6296c505a9a6dceea92c871fb629b71e76c92e911c2ba5d86 WatchSource:0}: Error finding container 24446ee291af9dd6296c505a9a6dceea92c871fb629b71e76c92e911c2ba5d86: Status 404 returned error can't find the container with id 24446ee291af9dd6296c505a9a6dceea92c871fb629b71e76c92e911c2ba5d86 Oct 30 19:21:23.819874 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:23.819841 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" event={"ID":"54643a49-f1da-4957-a6d6-81fb67991b2e","Type":"ContainerStarted","Data":"24446ee291af9dd6296c505a9a6dceea92c871fb629b71e76c92e911c2ba5d86"} Oct 30 19:21:23.821638 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:23.821613 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" event={"ID":"a12a3bdc-6b65-4496-8458-ff694a332732","Type":"ContainerStarted","Data":"b303640fedc1055469b55afce06b14c662a1a1028903a5df44fab6d3c6abcc91"} Oct 30 19:21:23.821747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:23.821735 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 19:21:24.708851 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.708786 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" podStartSLOduration=3.306837767 podStartE2EDuration="3.708767067s" totalImagesPullingTime="401.9293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:23.87646681 +0000 UTC m=+1036.950922171" watchObservedRunningTime="2026-10-30 19:21:24.708767067 +0000 UTC m=+1037.783222426" Oct 30 19:21:24.710264 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.710241 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d"] Oct 30 19:21:24.715187 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.714967 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.748377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.747526 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c5449c-openshift-default-dockercfg-k6524\"" Oct 30 19:21:24.767429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.766773 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.767429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.767061 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.767429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.767103 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn7v8\" (UniqueName: \"kubernetes.io/projected/10f19076-8c2f-4026-8c4b-ef3062bbe51f-kube-api-access-gn7v8\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.767429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.767214 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.767429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.767266 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.767429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.767394 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.767936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.767437 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.767936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.767480 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.767936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.767705 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.782188 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.780239 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d"] Oct 30 19:21:24.898476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.888046 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.898476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.888103 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn7v8\" (UniqueName: \"kubernetes.io/projected/10f19076-8c2f-4026-8c4b-ef3062bbe51f-kube-api-access-gn7v8\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.898476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.888149 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.898476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.888188 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.898476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.888243 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.898476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.888277 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.898476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.888306 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.898476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.888379 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.898476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.888450 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.898476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.889291 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.898476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.897998 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" event={"ID":"54643a49-f1da-4957-a6d6-81fb67991b2e","Type":"ContainerStarted","Data":"d4f10ce7955b30baf1793c139958d87144f98cd7b50aceeaa1313f688da0cf46"} Oct 30 19:21:24.898476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.898079 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" Oct 30 19:21:24.899484 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.899407 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.899705 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.899677 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.900143 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.900121 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.900578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.900497 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.904747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.903913 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.915038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.910935 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.919174 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.919143 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn7v8\" (UniqueName: \"kubernetes.io/projected/10f19076-8c2f-4026-8c4b-ef3062bbe51f-kube-api-access-gn7v8\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.919343 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.919310 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:24.966092 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:24.960298 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" podStartSLOduration=1.960275883 podStartE2EDuration="1.960275883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:24.944604035 +0000 UTC m=+1038.019059396" watchObservedRunningTime="2026-10-30 19:21:24.960275883 +0000 UTC m=+1038.034731242" Oct 30 19:21:25.040167 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.032397 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm"] Oct 30 19:21:25.040167 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.035931 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:25.058528 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.052322 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:21:25.080491 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.080448 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm"] Oct 30 19:21:25.085520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.084942 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-6e0c55-serving-cert\"" Oct 30 19:21:25.191768 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.191731 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq7r7\" (UniqueName: \"kubernetes.io/projected/2615a557-d1fd-4fe2-b23f-2785dd623271-kube-api-access-fq7r7\") pod \"maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm\" (UID: \"2615a557-d1fd-4fe2-b23f-2785dd623271\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:21:25.191919 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.191865 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2615a557-d1fd-4fe2-b23f-2785dd623271-maas-api-tls\") pod \"maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm\" (UID: \"2615a557-d1fd-4fe2-b23f-2785dd623271\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:21:25.191919 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.191893 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2615a557-d1fd-4fe2-b23f-2785dd623271-metrics-certs\") pod \"maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm\" (UID: \"2615a557-d1fd-4fe2-b23f-2785dd623271\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:21:25.192107 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.192081 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2615a557-d1fd-4fe2-b23f-2785dd623271-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm\" (UID: \"2615a557-d1fd-4fe2-b23f-2785dd623271\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:21:25.293752 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.293662 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq7r7\" (UniqueName: \"kubernetes.io/projected/2615a557-d1fd-4fe2-b23f-2785dd623271-kube-api-access-fq7r7\") pod \"maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm\" (UID: \"2615a557-d1fd-4fe2-b23f-2785dd623271\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:21:25.293949 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.293756 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2615a557-d1fd-4fe2-b23f-2785dd623271-maas-api-tls\") pod \"maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm\" (UID: \"2615a557-d1fd-4fe2-b23f-2785dd623271\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:21:25.293949 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.293786 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2615a557-d1fd-4fe2-b23f-2785dd623271-metrics-certs\") pod \"maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm\" (UID: \"2615a557-d1fd-4fe2-b23f-2785dd623271\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:21:25.293949 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.293896 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2615a557-d1fd-4fe2-b23f-2785dd623271-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm\" (UID: \"2615a557-d1fd-4fe2-b23f-2785dd623271\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:21:25.294628 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.294600 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2615a557-d1fd-4fe2-b23f-2785dd623271-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm\" (UID: \"2615a557-d1fd-4fe2-b23f-2785dd623271\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:21:25.296439 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.296404 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2615a557-d1fd-4fe2-b23f-2785dd623271-metrics-certs\") pod \"maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm\" (UID: \"2615a557-d1fd-4fe2-b23f-2785dd623271\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:21:25.296534 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.296458 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2615a557-d1fd-4fe2-b23f-2785dd623271-maas-api-tls\") pod \"maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm\" (UID: \"2615a557-d1fd-4fe2-b23f-2785dd623271\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:21:25.312434 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.312408 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq7r7\" (UniqueName: \"kubernetes.io/projected/2615a557-d1fd-4fe2-b23f-2785dd623271-kube-api-access-fq7r7\") pod \"maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm\" (UID: \"2615a557-d1fd-4fe2-b23f-2785dd623271\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:21:25.406495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.406460 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:21:25.425798 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.425768 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d"] Oct 30 19:21:25.427892 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:21:25.427854 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f19076_8c2f_4026_8c4b_ef3062bbe51f.slice/crio-4131ab6faf6b159c39a768891370e829587da882d8a2a11559c72ef36945c72e WatchSource:0}: Error finding container 4131ab6faf6b159c39a768891370e829587da882d8a2a11559c72ef36945c72e: Status 404 returned error can't find the container with id 4131ab6faf6b159c39a768891370e829587da882d8a2a11559c72ef36945c72e Oct 30 19:21:25.430434 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.430403 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:25.430530 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.430469 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:25.430530 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.430500 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:25.739031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.734487 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm"] Oct 30 19:21:25.903977 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.903935 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" event={"ID":"10f19076-8c2f-4026-8c4b-ef3062bbe51f","Type":"ContainerStarted","Data":"b9155f826cdcab0b00b8b1c444246c9bfcfacb4e1b98fe5eb6da89f216f69de5"} Oct 30 19:21:25.904499 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.904020 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" event={"ID":"10f19076-8c2f-4026-8c4b-ef3062bbe51f","Type":"ContainerStarted","Data":"4131ab6faf6b159c39a768891370e829587da882d8a2a11559c72ef36945c72e"} Oct 30 19:21:25.905448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:25.905414 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" event={"ID":"2615a557-d1fd-4fe2-b23f-2785dd623271","Type":"ContainerStarted","Data":"b1689c97300df86391f8b9e193a26a1ef37c6f9be99e4967e39aaed9fdc9a740"} Oct 30 19:21:26.058031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:26.057077 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:26.090624 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:26.090354 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:26.133318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:26.133262 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" podStartSLOduration=2.13324762 podStartE2EDuration="2.13324762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:25.949499393 +0000 UTC m=+1039.023954781" watchObservedRunningTime="2026-10-30 19:21:26.13324762 +0000 UTC m=+1039.207702976" Oct 30 19:21:26.927187 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:26.922428 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" event={"ID":"2615a557-d1fd-4fe2-b23f-2785dd623271","Type":"ContainerStarted","Data":"f3835b0b42ee438b352303509f756b2defd071b79de4790bf037fa83a9b5d90f"} Oct 30 19:21:26.927187 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:26.922928 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:21:26.927187 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:26.923473 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:26.939033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:26.934134 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:21:26.970072 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:26.963469 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" podStartSLOduration=1.5616677989999999 podStartE2EDuration="1.963448462s" totalImagesPullingTime="401.780663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:26.963263177 +0000 UTC m=+1040.037718539" watchObservedRunningTime="2026-10-30 19:21:26.963448462 +0000 UTC m=+1040.037903824" Oct 30 19:21:27.246089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.242733 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6"] Oct 30 19:21:27.265277 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.263716 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6"] Oct 30 19:21:27.265277 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.263839 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 19:21:27.278703 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.278205 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-5ac614-serving-cert\"" Oct 30 19:21:27.319692 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.319654 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6\" (UID: \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 19:21:27.319870 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.319721 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjq6p\" (UniqueName: \"kubernetes.io/projected/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-kube-api-access-fjq6p\") pod \"maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6\" (UID: \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 19:21:27.319870 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.319794 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-metrics-certs\") pod \"maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6\" (UID: \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 19:21:27.319870 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.319836 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6\" (UID: \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 19:21:27.420453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.420414 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6\" (UID: \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 19:21:27.420671 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.420479 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjq6p\" (UniqueName: \"kubernetes.io/projected/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-kube-api-access-fjq6p\") pod \"maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6\" (UID: \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 19:21:27.420671 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.420522 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-metrics-certs\") pod \"maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6\" (UID: \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 19:21:27.420671 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.420574 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6\" (UID: \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 19:21:27.421311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.421284 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6\" (UID: \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 19:21:27.423175 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.423136 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6\" (UID: \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 19:21:27.423306 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.423264 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-metrics-certs\") pod \"maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6\" (UID: \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 19:21:27.461796 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.461755 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjq6p\" (UniqueName: \"kubernetes.io/projected/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-kube-api-access-fjq6p\") pod \"maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6\" (UID: \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 19:21:27.583477 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.583427 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 19:21:27.913375 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.913344 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6"] Oct 30 19:21:27.915548 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:21:27.915514 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45e50f1_ccb5_4cdc_bd7b_75d85074b002.slice/crio-98c37ea3c24643f9c001bc91b6af9ed70ac0ea68eff1cc213c2cef2d3c2d6605 WatchSource:0}: Error finding container 98c37ea3c24643f9c001bc91b6af9ed70ac0ea68eff1cc213c2cef2d3c2d6605: Status 404 returned error can't find the container with id 98c37ea3c24643f9c001bc91b6af9ed70ac0ea68eff1cc213c2cef2d3c2d6605 Oct 30 19:21:27.929080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:27.929042 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" event={"ID":"e45e50f1-ccb5-4cdc-bd7b-75d85074b002","Type":"ContainerStarted","Data":"98c37ea3c24643f9c001bc91b6af9ed70ac0ea68eff1cc213c2cef2d3c2d6605"} Oct 30 19:21:28.962457 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:28.962404 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" event={"ID":"e45e50f1-ccb5-4cdc-bd7b-75d85074b002","Type":"ContainerStarted","Data":"593f43a62fb8913eb5a0146559b68fce4aa948e2892e537b566a58156dfe9cf9"} Oct 30 19:21:28.962967 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:28.962664 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 19:21:29.003682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:29.003618 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" podStartSLOduration=1.632354749 podStartE2EDuration="2.003601875s" totalImagesPullingTime="371.247126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:29.002887975 +0000 UTC m=+1042.077343335" watchObservedRunningTime="2026-10-30 19:21:29.003601875 +0000 UTC m=+1042.078057241" Oct 30 19:21:29.903701 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:29.903671 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 19:21:32.817585 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:32.817554 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" Oct 30 19:21:32.960232 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:32.960196 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:21:33.825200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:33.825158 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:21:33.825707 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:33.825242 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" podUID="9e15d741-48fd-4962-90db-de0e6cb664f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:21:34.978591 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:34.978565 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 19:21:35.006092 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:35.006061 5047 generic.go:356] "Generic (PLEG): container finished" podID="9e15d741-48fd-4962-90db-de0e6cb664f3" containerID="7b4274f5e71fc6a82355b76c97e5554c897e19ce89d3a0c939945db10ff1e3e2" exitCode=0 Oct 30 19:21:35.006256 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:35.006137 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" event={"ID":"9e15d741-48fd-4962-90db-de0e6cb664f3","Type":"ContainerDied","Data":"7b4274f5e71fc6a82355b76c97e5554c897e19ce89d3a0c939945db10ff1e3e2"} Oct 30 19:21:35.006532 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:35.006518 5047 scope.go:122] "RemoveContainer" containerID="7b4274f5e71fc6a82355b76c97e5554c897e19ce89d3a0c939945db10ff1e3e2" Oct 30 19:21:35.007920 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:35.007890 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:35.007979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:35.007965 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:35.008045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:35.008032 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:35.830454 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:35.830422 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm"] Oct 30 19:21:35.833769 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:35.833747 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 19:21:35.880977 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:35.877295 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-e67b84-serving-cert\"" Oct 30 19:21:35.882150 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:35.882119 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm"] Oct 30 19:21:35.929123 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:35.929084 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxsck\" (UniqueName: \"kubernetes.io/projected/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-kube-api-access-vxsck\") pod \"maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm\" (UID: \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 19:21:35.929408 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:35.929391 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm\" (UID: \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 19:21:35.929546 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:35.929535 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-metrics-certs\") pod \"maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm\" (UID: \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 19:21:35.929719 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:35.929708 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm\" (UID: \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 19:21:36.013906 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:36.013869 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" event={"ID":"9e15d741-48fd-4962-90db-de0e6cb664f3","Type":"ContainerStarted","Data":"7a12cf1a6ddf89b7ea2eec687c46087565bf565148655daeefb226bf951ad56d"} Oct 30 19:21:36.030850 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:36.030800 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-metrics-certs\") pod \"maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm\" (UID: \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 19:21:36.031169 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:36.031129 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm\" (UID: \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 19:21:36.031358 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:36.031333 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxsck\" (UniqueName: \"kubernetes.io/projected/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-kube-api-access-vxsck\") pod \"maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm\" (UID: \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 19:21:36.031427 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:36.031404 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm\" (UID: \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 19:21:36.031546 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:21:36.031524 5047 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-serving-cert: secret "maas-api-e2e-worker-w5-e67b84-serving-cert" not found Oct 30 19:21:36.031642 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:21:36.031596 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-maas-api-tls podName:d0fa5318-4e5f-4970-ac2f-39a8e839b6a5 nodeName:}" failed. No retries permitted until 2026-10-30 19:21:36.531576704 +0000 UTC m=+1049.606032055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-maas-api-tls") pod "maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" (UID: "d0fa5318-4e5f-4970-ac2f-39a8e839b6a5") : secret "maas-api-e2e-worker-w5-e67b84-serving-cert" not found Oct 30 19:21:36.032173 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:36.032147 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm\" (UID: \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 19:21:36.076091 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:36.075998 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxsck\" (UniqueName: \"kubernetes.io/projected/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-kube-api-access-vxsck\") pod \"maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm\" (UID: \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 19:21:36.098098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:36.097922 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-metrics-certs\") pod \"maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm\" (UID: \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 19:21:36.569084 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:36.567618 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm\" (UID: \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 19:21:36.571868 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:36.571836 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm\" (UID: \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 19:21:36.829598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:36.829571 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 19:21:36.891987 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:36.891959 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:36.892183 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:36.892036 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 19:21:36.892321 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:36.892304 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" podUID="9e15d741-48fd-4962-90db-de0e6cb664f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 19:21:36.926878 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:36.926851 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" Oct 30 19:21:37.193456 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:37.193427 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm"] Oct 30 19:21:37.198817 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:21:37.198785 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0fa5318_4e5f_4970_ac2f_39a8e839b6a5.slice/crio-2aaea9d86c98d83634fc5e2bb7c637560c4f53a9cab83d4d1fdf980659f6a531 WatchSource:0}: Error finding container 2aaea9d86c98d83634fc5e2bb7c637560c4f53a9cab83d4d1fdf980659f6a531: Status 404 returned error can't find the container with id 2aaea9d86c98d83634fc5e2bb7c637560c4f53a9cab83d4d1fdf980659f6a531 Oct 30 19:21:37.892073 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:37.892034 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 19:21:37.892257 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:37.892094 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" podUID="9e15d741-48fd-4962-90db-de0e6cb664f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 19:21:38.069033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:38.046995 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" event={"ID":"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5","Type":"ContainerStarted","Data":"f8167ff50861b58c8a9acd647ba50fa8641cae0ee3767791bd4b48c3a420f646"} Oct 30 19:21:38.069033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:38.047122 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" event={"ID":"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5","Type":"ContainerStarted","Data":"2aaea9d86c98d83634fc5e2bb7c637560c4f53a9cab83d4d1fdf980659f6a531"} Oct 30 19:21:38.069033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:38.048295 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 19:21:38.090975 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:38.090915 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" podStartSLOduration=2.7157399030000002 podStartE2EDuration="3.090899752s" totalImagesPullingTime="375.159849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:38.088507285 +0000 UTC m=+1051.162962645" watchObservedRunningTime="2026-10-30 19:21:38.090899752 +0000 UTC m=+1051.165355110" Oct 30 19:21:38.896276 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:38.892309 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 19:21:38.896276 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:38.892375 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" podUID="9e15d741-48fd-4962-90db-de0e6cb664f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 19:21:39.607451 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:21:39.607406 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:21:39.815036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:39.814971 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59"] Oct 30 19:21:39.818893 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:39.818866 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:21:39.845236 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:39.845160 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-249dd4-serving-cert\"" Oct 30 19:21:39.874706 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:39.874616 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59"] Oct 30 19:21:39.905246 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:39.905217 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 19:21:39.905703 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:39.905672 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" podUID="9e15d741-48fd-4962-90db-de0e6cb664f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 19:21:39.973453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:39.973417 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-metrics-certs\") pod \"maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59\" (UID: \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:21:39.973639 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:39.973585 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59\" (UID: \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:21:39.973691 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:39.973654 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksfg5\" (UniqueName: \"kubernetes.io/projected/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-kube-api-access-ksfg5\") pod \"maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59\" (UID: \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:21:39.973733 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:39.973712 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-maas-api-tls\") pod \"maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59\" (UID: \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:21:40.106147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:40.089599 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-maas-api-tls\") pod \"maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59\" (UID: \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:21:40.106462 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:40.106425 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-metrics-certs\") pod \"maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59\" (UID: \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:21:40.106695 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:40.106666 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59\" (UID: \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:21:40.106781 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:40.106772 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksfg5\" (UniqueName: \"kubernetes.io/projected/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-kube-api-access-ksfg5\") pod \"maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59\" (UID: \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:21:40.108271 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:40.108212 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59\" (UID: \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:21:40.116879 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:40.116850 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-metrics-certs\") pod \"maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59\" (UID: \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:21:40.126404 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:40.126277 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksfg5\" (UniqueName: \"kubernetes.io/projected/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-kube-api-access-ksfg5\") pod \"maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59\" (UID: \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:21:40.127934 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:40.127902 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-maas-api-tls\") pod \"maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59\" (UID: \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:21:40.133921 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:40.133705 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:21:40.490764 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:40.490580 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59"] Oct 30 19:21:40.493575 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:21:40.493541 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285e5b6a_ed6d_437d_ad30_7f6d2d7b9d57.slice/crio-89493e4f72a35dd9dcd82c381a62933e347812f149eb6a33b74a9d8d7f5cfb92 WatchSource:0}: Error finding container 89493e4f72a35dd9dcd82c381a62933e347812f149eb6a33b74a9d8d7f5cfb92: Status 404 returned error can't find the container with id 89493e4f72a35dd9dcd82c381a62933e347812f149eb6a33b74a9d8d7f5cfb92 Oct 30 19:21:40.892849 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:40.892803 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 19:21:40.893057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:40.892867 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" podUID="9e15d741-48fd-4962-90db-de0e6cb664f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 19:21:40.917748 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:21:40.915878 5047 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285e5b6a_ed6d_437d_ad30_7f6d2d7b9d57.slice/crio-7b5d37650c29e1b383e1e05eedc0bb435e9e8672512ee7bf60cba09f7ec92e2e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:21:41.117787 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:41.117702 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" event={"ID":"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57","Type":"ContainerStarted","Data":"7b5d37650c29e1b383e1e05eedc0bb435e9e8672512ee7bf60cba09f7ec92e2e"} Oct 30 19:21:41.117787 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:41.117766 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" event={"ID":"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57","Type":"ContainerStarted","Data":"89493e4f72a35dd9dcd82c381a62933e347812f149eb6a33b74a9d8d7f5cfb92"} Oct 30 19:21:41.118090 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:41.117974 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:21:42.125326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:42.125235 5047 generic.go:356] "Generic (PLEG): container finished" podID="9aff363b-dce0-48d0-91cc-7783972b1f85" containerID="919f6e2b4a4060e86af745524d5294dd59dabb80f1caf4d3d772f1155fb2a80f" exitCode=0 Oct 30 19:21:42.125326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:42.125311 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" event={"ID":"9aff363b-dce0-48d0-91cc-7783972b1f85","Type":"ContainerDied","Data":"919f6e2b4a4060e86af745524d5294dd59dabb80f1caf4d3d772f1155fb2a80f"} Oct 30 19:21:42.125916 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:42.125828 5047 scope.go:122] "RemoveContainer" containerID="919f6e2b4a4060e86af745524d5294dd59dabb80f1caf4d3d772f1155fb2a80f" Oct 30 19:21:42.127251 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:42.127218 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:42.127357 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:42.127306 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:42.127421 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:42.127357 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:42.171888 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:42.171049 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" podStartSLOduration=2.833150688 podStartE2EDuration="3.171027424s" totalImagesPullingTime="337.876736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:41.160345759 +0000 UTC m=+1054.234801118" watchObservedRunningTime="2026-10-30 19:21:42.171027424 +0000 UTC m=+1055.245482784" Oct 30 19:21:42.290203 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:42.290155 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:42.290899 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:42.290850 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:42.292354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:42.292299 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:21:43.133071 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:43.133040 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" event={"ID":"9aff363b-dce0-48d0-91cc-7783972b1f85","Type":"ContainerStarted","Data":"5ba6556e27f3d93cb3109c00b6c39811f725ec1ac8c94a2568b21e7bb0830ed6"} Oct 30 19:21:43.433526 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:43.433440 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:43.443030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:43.434881 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 19:21:43.443030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:43.434938 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" podUID="9aff363b-dce0-48d0-91cc-7783972b1f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 19:21:44.433268 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:44.433232 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 19:21:44.433674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:44.433299 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" podUID="9aff363b-dce0-48d0-91cc-7783972b1f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 19:21:45.122227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:45.122183 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 19:21:45.434374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:45.434281 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 19:21:45.435312 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:45.435273 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" podUID="9aff363b-dce0-48d0-91cc-7783972b1f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 19:21:46.444031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:46.438119 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 19:21:46.444031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:46.438204 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" podUID="9aff363b-dce0-48d0-91cc-7783972b1f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 19:21:47.131036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:47.130996 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:21:47.433344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:47.433253 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 19:21:47.433344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:47.433329 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" podUID="9aff363b-dce0-48d0-91cc-7783972b1f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 19:21:47.789096 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:47.787889 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:48.653823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:48.653476 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:48.668588 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:48.668555 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:21:57.831097 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:57.830971 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": read tcp 10.132.0.2:36160->10.132.0.60:15021: read: connection reset by peer" start-of-body= Oct 30 19:21:57.831527 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:57.831184 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" podUID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": read tcp 10.132.0.2:36160->10.132.0.60:15021: read: connection reset by peer" Oct 30 19:21:58.256750 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:58.256659 5047 generic.go:356] "Generic (PLEG): container finished" podID="8fd1d5de-7f02-4d79-a762-ee3d77c86bba" containerID="56763c4475020b081e4a70044ea983e9e32616c596850747719576485bb8d11c" exitCode=0 Oct 30 19:21:58.256750 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:58.256725 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" event={"ID":"8fd1d5de-7f02-4d79-a762-ee3d77c86bba","Type":"ContainerDied","Data":"56763c4475020b081e4a70044ea983e9e32616c596850747719576485bb8d11c"} Oct 30 19:21:58.257219 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:58.257200 5047 scope.go:122] "RemoveContainer" containerID="56763c4475020b081e4a70044ea983e9e32616c596850747719576485bb8d11c" Oct 30 19:21:58.258733 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:58.258710 5047 generic.go:356] "Generic (PLEG): container finished" podID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" containerID="b9155f826cdcab0b00b8b1c444246c9bfcfacb4e1b98fe5eb6da89f216f69de5" exitCode=0 Oct 30 19:21:58.258733 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:58.258716 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:58.258897 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:58.258781 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" event={"ID":"10f19076-8c2f-4026-8c4b-ef3062bbe51f","Type":"ContainerDied","Data":"b9155f826cdcab0b00b8b1c444246c9bfcfacb4e1b98fe5eb6da89f216f69de5"} Oct 30 19:21:58.258897 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:58.258800 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:58.258897 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:58.258851 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:58.259226 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:58.259205 5047 scope.go:122] "RemoveContainer" containerID="b9155f826cdcab0b00b8b1c444246c9bfcfacb4e1b98fe5eb6da89f216f69de5" Oct 30 19:21:58.260320 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:58.260293 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:58.260378 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:58.260357 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:58.260421 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:58.260394 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:59.283221 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:59.283171 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" event={"ID":"8fd1d5de-7f02-4d79-a762-ee3d77c86bba","Type":"ContainerStarted","Data":"4efff2cf40c7ba3c3e1463b96984fbe11730c195277fec9aa0b96347107d893b"} Oct 30 19:21:59.288202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:21:59.288168 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" event={"ID":"10f19076-8c2f-4026-8c4b-ef3062bbe51f","Type":"ContainerStarted","Data":"30226ee388745ce9ca41c8ccde0c1d49eaf56bc89387a7df0abbf1aa3a6ce660"} Oct 30 19:22:00.062030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:00.044039 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:22:00.089739 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:00.087894 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:22:00.089739 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:00.087958 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" podUID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 19:22:00.155054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:00.152705 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:22:00.155054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:00.152902 5047 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 19:22:00.155054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:00.152951 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" podUID="8fd1d5de-7f02-4d79-a762-ee3d77c86bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 19:22:01.043439 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:01.043402 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:22:01.044043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:01.043991 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" podUID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 19:22:01.167032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:01.166438 5047 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 19:22:01.167032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:01.166502 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" podUID="8fd1d5de-7f02-4d79-a762-ee3d77c86bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 19:22:01.787028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:01.786150 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:22:02.048091 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:02.042523 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:22:02.048091 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:02.042588 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" podUID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 19:22:02.156032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:02.155469 5047 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 19:22:02.156032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:02.155537 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" podUID="8fd1d5de-7f02-4d79-a762-ee3d77c86bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 19:22:03.045027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:03.044199 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:22:03.045027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:03.044262 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" podUID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 19:22:03.179073 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:03.177139 5047 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 19:22:03.179073 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:03.177220 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" podUID="8fd1d5de-7f02-4d79-a762-ee3d77c86bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 19:22:04.042470 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:04.042432 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:22:04.042788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:04.042759 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" podUID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 19:22:04.154032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:04.152557 5047 patch_prober.go:29] interesting pod/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 19:22:04.154032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:04.152648 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" podUID="8fd1d5de-7f02-4d79-a762-ee3d77c86bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 19:22:05.049021 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:05.048983 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:22:05.049560 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:05.049313 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:22:05.050145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:05.050127 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:22:05.154087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:05.154058 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:22:05.155438 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:05.155408 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:22:06.317218 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:06.317185 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-4sdcp-kxxdm"] Oct 30 19:22:06.322067 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:06.322044 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" Oct 30 19:22:06.337520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:06.337492 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4sdcp-kxxdm"] Oct 30 19:22:06.437814 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:06.437779 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/770e23c1-db57-4590-a46d-77ff7f9909ea-trusted-ca-bundle\") pod \"periodic-gathering-4sdcp-kxxdm\" (UID: \"770e23c1-db57-4590-a46d-77ff7f9909ea\") " pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" Oct 30 19:22:06.438037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:06.437940 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/770e23c1-db57-4590-a46d-77ff7f9909ea-archives-path\") pod \"periodic-gathering-4sdcp-kxxdm\" (UID: \"770e23c1-db57-4590-a46d-77ff7f9909ea\") " pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" Oct 30 19:22:06.438037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:06.437985 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/770e23c1-db57-4590-a46d-77ff7f9909ea-service-ca-bundle\") pod \"periodic-gathering-4sdcp-kxxdm\" (UID: \"770e23c1-db57-4590-a46d-77ff7f9909ea\") " pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" Oct 30 19:22:06.438171 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:06.438132 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xzfz\" (UniqueName: \"kubernetes.io/projected/770e23c1-db57-4590-a46d-77ff7f9909ea-kube-api-access-6xzfz\") pod \"periodic-gathering-4sdcp-kxxdm\" (UID: \"770e23c1-db57-4590-a46d-77ff7f9909ea\") " pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" Oct 30 19:22:06.539448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:06.539412 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/770e23c1-db57-4590-a46d-77ff7f9909ea-archives-path\") pod \"periodic-gathering-4sdcp-kxxdm\" (UID: \"770e23c1-db57-4590-a46d-77ff7f9909ea\") " pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" Oct 30 19:22:06.539658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:06.539458 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/770e23c1-db57-4590-a46d-77ff7f9909ea-service-ca-bundle\") pod \"periodic-gathering-4sdcp-kxxdm\" (UID: \"770e23c1-db57-4590-a46d-77ff7f9909ea\") " pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" Oct 30 19:22:06.539658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:06.539499 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xzfz\" (UniqueName: \"kubernetes.io/projected/770e23c1-db57-4590-a46d-77ff7f9909ea-kube-api-access-6xzfz\") pod \"periodic-gathering-4sdcp-kxxdm\" (UID: \"770e23c1-db57-4590-a46d-77ff7f9909ea\") " pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" Oct 30 19:22:06.539658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:06.539544 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/770e23c1-db57-4590-a46d-77ff7f9909ea-trusted-ca-bundle\") pod \"periodic-gathering-4sdcp-kxxdm\" (UID: \"770e23c1-db57-4590-a46d-77ff7f9909ea\") " pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" Oct 30 19:22:06.539912 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:06.539886 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/770e23c1-db57-4590-a46d-77ff7f9909ea-archives-path\") pod \"periodic-gathering-4sdcp-kxxdm\" (UID: \"770e23c1-db57-4590-a46d-77ff7f9909ea\") " pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" Oct 30 19:22:06.540272 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:06.540249 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/770e23c1-db57-4590-a46d-77ff7f9909ea-service-ca-bundle\") pod \"periodic-gathering-4sdcp-kxxdm\" (UID: \"770e23c1-db57-4590-a46d-77ff7f9909ea\") " pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" Oct 30 19:22:06.540350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:06.540300 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/770e23c1-db57-4590-a46d-77ff7f9909ea-trusted-ca-bundle\") pod \"periodic-gathering-4sdcp-kxxdm\" (UID: \"770e23c1-db57-4590-a46d-77ff7f9909ea\") " pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" Oct 30 19:22:06.558494 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:06.558463 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xzfz\" (UniqueName: \"kubernetes.io/projected/770e23c1-db57-4590-a46d-77ff7f9909ea-kube-api-access-6xzfz\") pod \"periodic-gathering-4sdcp-kxxdm\" (UID: \"770e23c1-db57-4590-a46d-77ff7f9909ea\") " pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" Oct 30 19:22:06.637757 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:06.637718 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" Oct 30 19:22:06.940600 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:06.940567 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4sdcp-kxxdm"] Oct 30 19:22:06.942083 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:22:06.942045 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770e23c1_db57_4590_a46d_77ff7f9909ea.slice/crio-00c766d6b61e48285fe0bb0233de710bc15e541478ab5cc861f289bcffe8e2a6 WatchSource:0}: Error finding container 00c766d6b61e48285fe0bb0233de710bc15e541478ab5cc861f289bcffe8e2a6: Status 404 returned error can't find the container with id 00c766d6b61e48285fe0bb0233de710bc15e541478ab5cc861f289bcffe8e2a6 Oct 30 19:22:07.351459 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:07.351400 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" event={"ID":"770e23c1-db57-4590-a46d-77ff7f9909ea","Type":"ContainerStarted","Data":"03e994a45674125768333da1384a02817a178b39be5cc3b44ff537b59daf46e5"} Oct 30 19:22:07.351459 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:07.351464 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" event={"ID":"770e23c1-db57-4590-a46d-77ff7f9909ea","Type":"ContainerStarted","Data":"00c766d6b61e48285fe0bb0233de710bc15e541478ab5cc861f289bcffe8e2a6"} Oct 30 19:22:07.389616 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:07.389561 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" podStartSLOduration=1.38954565 podStartE2EDuration="1.38954565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:07.388885028 +0000 UTC m=+1080.463340386" watchObservedRunningTime="2026-10-30 19:22:07.38954565 +0000 UTC m=+1080.464001047" Oct 30 19:22:09.601650 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:22:09.601613 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:22:32.232205 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.232165 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt"] Oct 30 19:22:32.236498 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.236471 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.251681 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.251636 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5ac614\"/\"openshift-service-ca.crt\"" Oct 30 19:22:32.263046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.262990 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5ac614\"/\"kube-root-ca.crt\"" Oct 30 19:22:32.263214 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.263044 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5ac614\"/\"premium-w1-5ac614-kserve-self-signed-certs\"" Oct 30 19:22:32.263299 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.263147 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5ac614\"/\"default-dockercfg-2pfzk\"" Oct 30 19:22:32.265750 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.265726 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt"] Oct 30 19:22:32.349542 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.349511 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-home\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.349704 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.349572 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9325c7d7-408b-44dc-a660-9f15ec74f9e6-tls-certs\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.349752 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.349702 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-tmp-dir\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.349789 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.349754 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-model-cache\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.349955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.349926 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4krv5\" (UniqueName: \"kubernetes.io/projected/9325c7d7-408b-44dc-a660-9f15ec74f9e6-kube-api-access-4krv5\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.349999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.349983 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-dshm\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.455147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.454201 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-home\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.455147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.454290 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9325c7d7-408b-44dc-a660-9f15ec74f9e6-tls-certs\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.455147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.454341 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-tmp-dir\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.455147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.454381 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-model-cache\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.455147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.454487 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4krv5\" (UniqueName: \"kubernetes.io/projected/9325c7d7-408b-44dc-a660-9f15ec74f9e6-kube-api-access-4krv5\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.455147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.454524 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-dshm\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.488109 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.460556 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-tmp-dir\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.488109 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.460813 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-home\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.488109 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.461718 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-model-cache\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.488109 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.473050 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9325c7d7-408b-44dc-a660-9f15ec74f9e6-tls-certs\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.488109 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.477394 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-dshm\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.488839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.488807 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4krv5\" (UniqueName: \"kubernetes.io/projected/9325c7d7-408b-44dc-a660-9f15ec74f9e6-kube-api-access-4krv5\") pod \"premium-w1-5ac614-kserve-57fd77fb9f-swqjt\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.558679 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.558638 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:32.854907 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:32.854876 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt"] Oct 30 19:22:32.857042 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:22:32.856985 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9325c7d7_408b_44dc_a660_9f15ec74f9e6.slice/crio-4001b8c88687505fe0ffdf4aa80da59e0bd8a2b39597e053a22dbe108c18696a WatchSource:0}: Error finding container 4001b8c88687505fe0ffdf4aa80da59e0bd8a2b39597e053a22dbe108c18696a: Status 404 returned error can't find the container with id 4001b8c88687505fe0ffdf4aa80da59e0bd8a2b39597e053a22dbe108c18696a Oct 30 19:22:33.501170 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:33.501131 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" event={"ID":"9325c7d7-408b-44dc-a660-9f15ec74f9e6","Type":"ContainerStarted","Data":"63fc02478fe554df547be72df72c45574f0ffd4e645c3dc9d6bf0b4b4d930f72"} Oct 30 19:22:33.501614 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:33.501184 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" event={"ID":"9325c7d7-408b-44dc-a660-9f15ec74f9e6","Type":"ContainerStarted","Data":"4001b8c88687505fe0ffdf4aa80da59e0bd8a2b39597e053a22dbe108c18696a"} Oct 30 19:22:33.559466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:33.559424 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:39.601188 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:22:39.601143 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:22:39.993798 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:39.993684 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podStartSLOduration=7.993665662 podStartE2EDuration="7.993665662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:33.541200547 +0000 UTC m=+1106.615655908" watchObservedRunningTime="2026-10-30 19:22:39.993665662 +0000 UTC m=+1113.068121021" Oct 30 19:22:39.995144 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:39.995120 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq"] Oct 30 19:22:40.000755 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.000729 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.016342 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.016316 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w"] Oct 30 19:22:40.020218 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.020185 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.028779 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.028269 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq"] Oct 30 19:22:40.040935 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.040875 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w"] Oct 30 19:22:40.079779 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.079378 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7b1347-f416-42a3-a055-0b866e894c64-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.079779 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.079441 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.079779 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.079476 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.079779 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.079508 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7b1347-f416-42a3-a055-0b866e894c64-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.079779 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.079548 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.079779 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.079587 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.079779 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.079617 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.079779 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.079701 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tndr\" (UniqueName: \"kubernetes.io/projected/0d95b5e1-e759-430f-b0fb-461451ee6e32-kube-api-access-2tndr\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.079779 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.079733 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.079779 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.079758 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.080496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.079809 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.080496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.079848 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.080496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.079876 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.080496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.079916 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d95b5e1-e759-430f-b0fb-461451ee6e32-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.080496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.079945 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7b1347-f416-42a3-a055-0b866e894c64-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.080496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.079976 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66fg7\" (UniqueName: \"kubernetes.io/projected/5c7b1347-f416-42a3-a055-0b866e894c64-kube-api-access-66fg7\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.080496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.080019 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.080496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.080111 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.180796 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.180756 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d95b5e1-e759-430f-b0fb-461451ee6e32-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.180998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.180802 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7b1347-f416-42a3-a055-0b866e894c64-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.180998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.180831 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66fg7\" (UniqueName: \"kubernetes.io/projected/5c7b1347-f416-42a3-a055-0b866e894c64-kube-api-access-66fg7\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.180998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.180861 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.180998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.180912 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.180998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.180940 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7b1347-f416-42a3-a055-0b866e894c64-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.180998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.180973 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.181326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.180997 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.181326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.181073 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7b1347-f416-42a3-a055-0b866e894c64-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.181326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.181105 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.181326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.181134 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.181326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.181163 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.181326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.181245 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tndr\" (UniqueName: \"kubernetes.io/projected/0d95b5e1-e759-430f-b0fb-461451ee6e32-kube-api-access-2tndr\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.181326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.181270 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.181326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.181297 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.181722 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.181348 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.181722 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.181400 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.181722 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.181684 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.181722 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.181701 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d95b5e1-e759-430f-b0fb-461451ee6e32-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.181933 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.181754 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.181933 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.181354 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.181933 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.181828 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.181933 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.181866 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.182169 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.181939 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.182169 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.182077 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.182440 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.182193 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.182503 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.182456 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.182601 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.182543 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7b1347-f416-42a3-a055-0b866e894c64-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.183762 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.183731 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7b1347-f416-42a3-a055-0b866e894c64-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.184118 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.184092 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.184335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.184308 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.184608 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.184592 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.206245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.206209 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7b1347-f416-42a3-a055-0b866e894c64-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.206398 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.206224 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.206398 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.206318 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66fg7\" (UniqueName: \"kubernetes.io/projected/5c7b1347-f416-42a3-a055-0b866e894c64-kube-api-access-66fg7\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.206479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.206448 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tndr\" (UniqueName: \"kubernetes.io/projected/0d95b5e1-e759-430f-b0fb-461451ee6e32-kube-api-access-2tndr\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-db77w\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.315736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.315638 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:40.362862 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.362828 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:40.669758 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.669727 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq"] Oct 30 19:22:40.671726 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:22:40.671699 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7b1347_f416_42a3_a055_0b866e894c64.slice/crio-fdc6872fa52a9a30b3194d6fec0347d91321533150fff2ac0a384029f7a11989 WatchSource:0}: Error finding container fdc6872fa52a9a30b3194d6fec0347d91321533150fff2ac0a384029f7a11989: Status 404 returned error can't find the container with id fdc6872fa52a9a30b3194d6fec0347d91321533150fff2ac0a384029f7a11989 Oct 30 19:22:40.674151 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.674113 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:22:40.674218 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.674183 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:22:40.674253 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.674218 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:22:40.687576 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.687545 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w"] Oct 30 19:22:40.691786 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:22:40.691727 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d95b5e1_e759_430f_b0fb_461451ee6e32.slice/crio-ceda3082d97135895e9c13e0218aa3491bf75b0176124476a44e3aaebba5a3e8 WatchSource:0}: Error finding container ceda3082d97135895e9c13e0218aa3491bf75b0176124476a44e3aaebba5a3e8: Status 404 returned error can't find the container with id ceda3082d97135895e9c13e0218aa3491bf75b0176124476a44e3aaebba5a3e8 Oct 30 19:22:40.694392 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.694354 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:22:40.694464 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.694442 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:22:40.694508 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:40.694492 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:22:41.586407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:41.581864 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" event={"ID":"5c7b1347-f416-42a3-a055-0b866e894c64","Type":"ContainerStarted","Data":"acc4b7dbe508305dfd7caf239d9f7f91cbe2fd196d9cbe8263a0f8f34331fe7b"} Oct 30 19:22:41.586407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:41.581938 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" event={"ID":"5c7b1347-f416-42a3-a055-0b866e894c64","Type":"ContainerStarted","Data":"fdc6872fa52a9a30b3194d6fec0347d91321533150fff2ac0a384029f7a11989"} Oct 30 19:22:41.587094 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:41.587062 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" event={"ID":"0d95b5e1-e759-430f-b0fb-461451ee6e32","Type":"ContainerStarted","Data":"395101f7331fcfba80d06b4c1c99bcc05b0f3ff05bfaec8129452b83ccce20fc"} Oct 30 19:22:41.587288 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:41.587273 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" event={"ID":"0d95b5e1-e759-430f-b0fb-461451ee6e32","Type":"ContainerStarted","Data":"ceda3082d97135895e9c13e0218aa3491bf75b0176124476a44e3aaebba5a3e8"} Oct 30 19:22:41.654032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:41.649874 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" podStartSLOduration=2.649851469 podStartE2EDuration="2.649851469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:41.634488097 +0000 UTC m=+1114.708943456" watchObservedRunningTime="2026-10-30 19:22:41.649851469 +0000 UTC m=+1114.724306829" Oct 30 19:22:42.316294 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:42.316254 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:42.317507 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:42.317477 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:22:42.317654 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:42.317525 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" podUID="5c7b1347-f416-42a3-a055-0b866e894c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 19:22:42.369554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:42.365313 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:42.375028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:42.374179 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:22:42.375028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:42.374245 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" podUID="0d95b5e1-e759-430f-b0fb-461451ee6e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:22:42.565931 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:42.563389 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:42.595468 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:42.595441 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:42.606931 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:42.606892 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:22:42.638031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:42.635967 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" podStartSLOduration=3.635949093 podStartE2EDuration="3.635949093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:41.681458236 +0000 UTC m=+1114.755913590" watchObservedRunningTime="2026-10-30 19:22:42.635949093 +0000 UTC m=+1115.710404454" Oct 30 19:22:43.316944 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:43.316908 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:22:43.317358 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:43.316967 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" podUID="5c7b1347-f416-42a3-a055-0b866e894c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 19:22:43.368025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:43.366543 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:22:43.368025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:43.366637 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" podUID="0d95b5e1-e759-430f-b0fb-461451ee6e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:22:44.317030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:44.316988 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:22:44.317401 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:44.317048 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" podUID="5c7b1347-f416-42a3-a055-0b866e894c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 19:22:44.364439 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:44.364399 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:22:44.364764 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:44.364734 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" podUID="0d95b5e1-e759-430f-b0fb-461451ee6e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:22:45.316271 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:45.316235 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:22:45.316465 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:45.316292 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" podUID="5c7b1347-f416-42a3-a055-0b866e894c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 19:22:45.364738 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:45.364699 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:22:45.365377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:45.365343 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" podUID="0d95b5e1-e759-430f-b0fb-461451ee6e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:22:46.316332 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:46.316298 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:22:46.316548 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:46.316354 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" podUID="5c7b1347-f416-42a3-a055-0b866e894c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 19:22:46.375032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:46.368199 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:22:46.375032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:46.368285 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" podUID="0d95b5e1-e759-430f-b0fb-461451ee6e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:22:47.353934 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:47.353895 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:47.355335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:47.355306 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:47.356632 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:47.356610 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:22:47.384573 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:47.384538 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:47.385098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:47.384984 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:22:47.386210 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:22:47.386184 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:23:00.309928 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.309889 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq"] Oct 30 19:23:00.314441 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.314415 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.329164 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.329143 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-6e0c55-openshift-default-dockercfg-mndx2\"" Oct 30 19:23:00.344663 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.344631 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq"] Oct 30 19:23:00.392914 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.392871 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.393128 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.393032 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.393128 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.393104 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.393255 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.393200 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7abf871e-98f2-4f49-9660-4e5640c12a1a-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.393300 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.393255 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm7rk\" (UniqueName: \"kubernetes.io/projected/7abf871e-98f2-4f49-9660-4e5640c12a1a-kube-api-access-fm7rk\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.393358 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.393345 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.393393 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.393366 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.393393 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.393382 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.393455 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.393400 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.494836 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.494794 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.495040 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.494880 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.495040 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.494930 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7abf871e-98f2-4f49-9660-4e5640c12a1a-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.495040 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.494980 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm7rk\" (UniqueName: \"kubernetes.io/projected/7abf871e-98f2-4f49-9660-4e5640c12a1a-kube-api-access-fm7rk\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.495223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.495045 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.495223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.495073 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.495223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.495095 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.495223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.495125 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.495419 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.495226 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.495419 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.495291 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.495419 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.495359 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.495580 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.495443 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.495712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.495686 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7abf871e-98f2-4f49-9660-4e5640c12a1a-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.495897 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.495872 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.498245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.498217 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.500525 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.500504 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.514513 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.514484 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.515381 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.515353 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm7rk\" (UniqueName: \"kubernetes.io/projected/7abf871e-98f2-4f49-9660-4e5640c12a1a-kube-api-access-fm7rk\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.628070 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.627995 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:00.946181 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.946153 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq"] Oct 30 19:23:00.947401 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:23:00.947376 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abf871e_98f2_4f49_9660_4e5640c12a1a.slice/crio-47bf179e9aee8ce61e76c6856f235bcceec3bb48d833c79d31d0399428d6eb74 WatchSource:0}: Error finding container 47bf179e9aee8ce61e76c6856f235bcceec3bb48d833c79d31d0399428d6eb74: Status 404 returned error can't find the container with id 47bf179e9aee8ce61e76c6856f235bcceec3bb48d833c79d31d0399428d6eb74 Oct 30 19:23:00.949844 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.949809 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:00.949936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.949876 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:00.949936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:00.949904 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:01.716889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:01.716849 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" event={"ID":"7abf871e-98f2-4f49-9660-4e5640c12a1a","Type":"ContainerStarted","Data":"7c57e8e496e4930ced53cb41f5d0e33ebadfd6d82d11ecacd30d0c320b4edcac"} Oct 30 19:23:01.717428 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:01.716933 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" event={"ID":"7abf871e-98f2-4f49-9660-4e5640c12a1a","Type":"ContainerStarted","Data":"47bf179e9aee8ce61e76c6856f235bcceec3bb48d833c79d31d0399428d6eb74"} Oct 30 19:23:01.791653 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:01.791586 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" podStartSLOduration=1.791568346 podStartE2EDuration="1.791568346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:01.788780456 +0000 UTC m=+1134.863235815" watchObservedRunningTime="2026-10-30 19:23:01.791568346 +0000 UTC m=+1134.866023704" Oct 30 19:23:02.628802 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:02.628743 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:02.630532 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:02.630500 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:23:02.630666 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:02.630558 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" podUID="7abf871e-98f2-4f49-9660-4e5640c12a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:23:03.707028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:03.706916 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:23:03.707028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:03.706989 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" podUID="7abf871e-98f2-4f49-9660-4e5640c12a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:23:04.628881 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:04.628836 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:23:04.629117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:04.628909 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" podUID="7abf871e-98f2-4f49-9660-4e5640c12a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:23:05.629355 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:05.629318 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:23:05.629829 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:05.629379 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" podUID="7abf871e-98f2-4f49-9660-4e5640c12a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:23:06.629174 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:06.629138 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:23:06.629362 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:06.629195 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" podUID="7abf871e-98f2-4f49-9660-4e5640c12a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:23:07.634111 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:07.631418 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:23:07.634111 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:07.631500 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" podUID="7abf871e-98f2-4f49-9660-4e5640c12a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:23:08.672988 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:08.672953 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:08.674883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:08.674861 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:08.676262 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:08.676238 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:23:09.601495 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:23:09.601456 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:23:15.125298 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.125259 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c"] Oct 30 19:23:15.130863 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.130842 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.150605 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.150571 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c"] Oct 30 19:23:15.158626 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.158595 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-e67b84-openshift-default-dockercfg-dvgvb\"" Oct 30 19:23:15.191618 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.191576 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.191618 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.191618 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zvkb\" (UniqueName: \"kubernetes.io/projected/8f5a1117-41f6-421e-a9cc-9230eea118f8-kube-api-access-9zvkb\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.191884 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.191739 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.191884 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.191794 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.191884 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.191812 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.192156 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.191916 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.192156 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.191965 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.192156 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.192018 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5a1117-41f6-421e-a9cc-9230eea118f8-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.192156 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.192072 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.293478 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.293440 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.293630 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.293491 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zvkb\" (UniqueName: \"kubernetes.io/projected/8f5a1117-41f6-421e-a9cc-9230eea118f8-kube-api-access-9zvkb\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.293630 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.293529 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.293630 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.293577 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.293630 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.293602 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.293828 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.293660 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.293828 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.293729 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.293828 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.293763 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5a1117-41f6-421e-a9cc-9230eea118f8-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.293828 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.293805 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.293981 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.293890 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.294464 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.294439 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.294594 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.294512 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.294594 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.294545 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5a1117-41f6-421e-a9cc-9230eea118f8-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.294675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.294616 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.296307 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.296282 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.296413 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.296332 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.312718 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.312691 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.313034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.312995 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zvkb\" (UniqueName: \"kubernetes.io/projected/8f5a1117-41f6-421e-a9cc-9230eea118f8-kube-api-access-9zvkb\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.444972 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.444888 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:15.757929 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.757896 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c"] Oct 30 19:23:15.759776 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:23:15.759745 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5a1117_41f6_421e_a9cc_9230eea118f8.slice/crio-e12531ba5fde50cd0efb1c500f70382a1f47b5613e063b61d4ef2eb2a167e75f WatchSource:0}: Error finding container e12531ba5fde50cd0efb1c500f70382a1f47b5613e063b61d4ef2eb2a167e75f: Status 404 returned error can't find the container with id e12531ba5fde50cd0efb1c500f70382a1f47b5613e063b61d4ef2eb2a167e75f Oct 30 19:23:15.762342 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.762310 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:15.762406 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.762375 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:15.762464 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.762406 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:15.848443 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:15.848412 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" event={"ID":"8f5a1117-41f6-421e-a9cc-9230eea118f8","Type":"ContainerStarted","Data":"e12531ba5fde50cd0efb1c500f70382a1f47b5613e063b61d4ef2eb2a167e75f"} Oct 30 19:23:16.932616 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:16.932570 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" event={"ID":"8f5a1117-41f6-421e-a9cc-9230eea118f8","Type":"ContainerStarted","Data":"0d1a88c3b7420c62a8b16df929c0c158f0354070793b1e28b579ad70ca58dc64"} Oct 30 19:23:16.994792 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:16.994723 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" podStartSLOduration=1.994703693 podStartE2EDuration="1.994703693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:16.992630169 +0000 UTC m=+1150.067085530" watchObservedRunningTime="2026-10-30 19:23:16.994703693 +0000 UTC m=+1150.069159055" Oct 30 19:23:17.449032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:17.445163 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:17.454029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:17.449675 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:23:17.454029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:17.449760 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" podUID="8f5a1117-41f6-421e-a9cc-9230eea118f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:23:18.459214 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:18.459171 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:23:18.459748 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:18.459237 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" podUID="8f5a1117-41f6-421e-a9cc-9230eea118f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:23:19.459229 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:19.459187 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:23:19.459755 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:19.459262 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" podUID="8f5a1117-41f6-421e-a9cc-9230eea118f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:23:20.456494 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:20.456448 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:23:20.456820 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:20.456788 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" podUID="8f5a1117-41f6-421e-a9cc-9230eea118f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:23:21.449451 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:21.449411 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:23:21.450087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:21.450053 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" podUID="8f5a1117-41f6-421e-a9cc-9230eea118f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:23:22.445920 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:22.445876 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:23:22.446109 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:22.445943 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" podUID="8f5a1117-41f6-421e-a9cc-9230eea118f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:23:23.449830 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:23.449794 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:23.450257 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:23.450097 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:23.450744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:23.450723 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:23:25.134255 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.134216 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv"] Oct 30 19:23:25.138093 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.138075 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.166702 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.166670 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv"] Oct 30 19:23:25.278795 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.278763 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.278984 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.278813 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2tc6\" (UniqueName: \"kubernetes.io/projected/04defd95-0e95-4ee6-9d47-d6b2488a2f62-kube-api-access-v2tc6\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.278984 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.278932 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.279108 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.278985 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.279108 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.279087 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.279173 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.279149 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.279223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.279211 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.279290 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.279276 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.279325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.279298 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.380617 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.380576 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.380802 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.380646 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.380802 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.380681 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.380802 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.380698 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.380802 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.380750 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.380802 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.380796 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tc6\" (UniqueName: \"kubernetes.io/projected/04defd95-0e95-4ee6-9d47-d6b2488a2f62-kube-api-access-v2tc6\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.381057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.380857 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.381057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.380907 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.381057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.380954 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.381274 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.381247 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-data\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.381384 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.381360 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-credential-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.381467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.381400 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-workload-socket\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.381510 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.381461 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-workload-certs\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.381559 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.381535 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istiod-ca-cert\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.383127 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.383108 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-envoy\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.383572 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.383552 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-podinfo\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.399708 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.399636 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-token\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.399945 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.399920 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2tc6\" (UniqueName: \"kubernetes.io/projected/04defd95-0e95-4ee6-9d47-d6b2488a2f62-kube-api-access-v2tc6\") pod \"e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.453284 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.453244 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:25.769516 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.769486 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv"] Oct 30 19:23:25.770342 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:23:25.770312 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04defd95_0e95_4ee6_9d47_d6b2488a2f62.slice/crio-1696e0932b28c1e787b8fa5c9b9bc2fe9df318f744d142e9846f0db708524f57 WatchSource:0}: Error finding container 1696e0932b28c1e787b8fa5c9b9bc2fe9df318f744d142e9846f0db708524f57: Status 404 returned error can't find the container with id 1696e0932b28c1e787b8fa5c9b9bc2fe9df318f744d142e9846f0db708524f57 Oct 30 19:23:25.772804 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.772772 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:25.772895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.772836 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:25.772895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:25.772863 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:26.011641 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:26.011598 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" event={"ID":"04defd95-0e95-4ee6-9d47-d6b2488a2f62","Type":"ContainerStarted","Data":"7e8adb1e3556cce603e59051efbb04508134441b93977ceaf6cb443730cbc7fd"} Oct 30 19:23:26.011837 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:26.011655 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" event={"ID":"04defd95-0e95-4ee6-9d47-d6b2488a2f62","Type":"ContainerStarted","Data":"1696e0932b28c1e787b8fa5c9b9bc2fe9df318f744d142e9846f0db708524f57"} Oct 30 19:23:26.056323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:26.056216 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" podStartSLOduration=1.05619631 podStartE2EDuration="1.05619631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:26.053992993 +0000 UTC m=+1159.128448354" watchObservedRunningTime="2026-10-30 19:23:26.05619631 +0000 UTC m=+1159.130651672" Oct 30 19:23:26.453741 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:26.453703 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:26.455130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:26.455106 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:23:26.455208 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:26.455170 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" podUID="04defd95-0e95-4ee6-9d47-d6b2488a2f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:23:27.459539 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:27.459502 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:23:27.459981 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:27.459563 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" podUID="04defd95-0e95-4ee6-9d47-d6b2488a2f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:23:28.459025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:28.458541 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:23:28.459025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:28.458616 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" podUID="04defd95-0e95-4ee6-9d47-d6b2488a2f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:23:29.462033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:29.456657 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:23:29.462033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:29.456728 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" podUID="04defd95-0e95-4ee6-9d47-d6b2488a2f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:23:30.459961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:30.459486 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:23:30.459961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:30.459549 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" podUID="04defd95-0e95-4ee6-9d47-d6b2488a2f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:23:31.458030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:31.457441 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:23:31.458030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:31.457512 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" podUID="04defd95-0e95-4ee6-9d47-d6b2488a2f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:23:32.454533 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:32.454496 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:23:32.454701 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:32.454571 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" podUID="04defd95-0e95-4ee6-9d47-d6b2488a2f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:23:34.005325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:34.005283 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:34.005958 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:34.005934 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:34.007048 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:34.006989 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:23:39.603252 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:23:39.603209 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:23:40.016957 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.016869 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44"] Oct 30 19:23:40.022465 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.022439 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.037895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.037859 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b97379-openshift-default-dockercfg-zcgdk\"" Oct 30 19:23:40.051786 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.051757 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44"] Oct 30 19:23:40.103061 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.102990 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709e2b00-e43d-425d-855f-8b498883c049-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.103257 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.103106 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709e2b00-e43d-425d-855f-8b498883c049-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.103257 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.103177 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.103370 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.103275 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwdnw\" (UniqueName: \"kubernetes.io/projected/709e2b00-e43d-425d-855f-8b498883c049-kube-api-access-kwdnw\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.103370 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.103316 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.103462 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.103441 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.103497 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.103486 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.103541 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.103505 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.103579 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.103567 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709e2b00-e43d-425d-855f-8b498883c049-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.204651 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.204606 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.204865 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.204675 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.204865 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.204701 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.204865 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.204734 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709e2b00-e43d-425d-855f-8b498883c049-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.204865 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.204831 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709e2b00-e43d-425d-855f-8b498883c049-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.205139 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.204869 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709e2b00-e43d-425d-855f-8b498883c049-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.205139 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.204946 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.205139 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.204984 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwdnw\" (UniqueName: \"kubernetes.io/projected/709e2b00-e43d-425d-855f-8b498883c049-kube-api-access-kwdnw\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.205139 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.205084 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.205328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.205146 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.205328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.205088 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.205328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.205191 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.205466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.205341 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.205705 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.205678 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709e2b00-e43d-425d-855f-8b498883c049-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.207444 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.207417 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.207681 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.207655 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709e2b00-e43d-425d-855f-8b498883c049-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.223657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.223617 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709e2b00-e43d-425d-855f-8b498883c049-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.223916 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.223894 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwdnw\" (UniqueName: \"kubernetes.io/projected/709e2b00-e43d-425d-855f-8b498883c049-kube-api-access-kwdnw\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.338217 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.338182 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:40.679636 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.673710 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44"] Oct 30 19:23:40.679636 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:23:40.678648 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709e2b00_e43d_425d_855f_8b498883c049.slice/crio-fc60f1bf8382f62ef8a4358c84b3d265e6181f99d80ba34f100dee6a8c463e81 WatchSource:0}: Error finding container fc60f1bf8382f62ef8a4358c84b3d265e6181f99d80ba34f100dee6a8c463e81: Status 404 returned error can't find the container with id fc60f1bf8382f62ef8a4358c84b3d265e6181f99d80ba34f100dee6a8c463e81 Oct 30 19:23:40.683851 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.683811 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:40.683988 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.683903 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:40.683988 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:40.683946 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:41.194363 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:41.194266 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" event={"ID":"709e2b00-e43d-425d-855f-8b498883c049","Type":"ContainerStarted","Data":"1d5e042c5872cac499956b583b10fad0b3cbf7ccdd639a38a914e0deaa7ea46a"} Oct 30 19:23:41.194363 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:41.194334 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" event={"ID":"709e2b00-e43d-425d-855f-8b498883c049","Type":"ContainerStarted","Data":"fc60f1bf8382f62ef8a4358c84b3d265e6181f99d80ba34f100dee6a8c463e81"} Oct 30 19:23:41.196081 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:41.196051 5047 generic.go:356] "Generic (PLEG): container finished" podID="770e23c1-db57-4590-a46d-77ff7f9909ea" containerID="03e994a45674125768333da1384a02817a178b39be5cc3b44ff537b59daf46e5" exitCode=0 Oct 30 19:23:41.196221 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:41.196102 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" event={"ID":"770e23c1-db57-4590-a46d-77ff7f9909ea","Type":"ContainerDied","Data":"03e994a45674125768333da1384a02817a178b39be5cc3b44ff537b59daf46e5"} Oct 30 19:23:41.272133 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:41.272074 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" podStartSLOduration=2.272057447 podStartE2EDuration="2.272057447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:41.259518877 +0000 UTC m=+1174.333974238" watchObservedRunningTime="2026-10-30 19:23:41.272057447 +0000 UTC m=+1174.346512805" Oct 30 19:23:41.338617 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:41.338534 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:41.347678 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:41.347505 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:23:41.347678 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:41.347584 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" podUID="709e2b00-e43d-425d-855f-8b498883c049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:23:42.340461 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:42.340420 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:23:42.341109 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:42.341067 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" podUID="709e2b00-e43d-425d-855f-8b498883c049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:23:43.314340 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.314237 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" Oct 30 19:23:43.370035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.342666 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:23:43.370035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.342739 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" podUID="709e2b00-e43d-425d-855f-8b498883c049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:23:43.393803 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.389047 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/770e23c1-db57-4590-a46d-77ff7f9909ea-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/770e23c1-db57-4590-a46d-77ff7f9909ea-service-ca-bundle\") pod \"770e23c1-db57-4590-a46d-77ff7f9909ea\" (UID: \"770e23c1-db57-4590-a46d-77ff7f9909ea\") " Oct 30 19:23:43.393803 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.389232 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/770e23c1-db57-4590-a46d-77ff7f9909ea-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/770e23c1-db57-4590-a46d-77ff7f9909ea-trusted-ca-bundle\") pod \"770e23c1-db57-4590-a46d-77ff7f9909ea\" (UID: \"770e23c1-db57-4590-a46d-77ff7f9909ea\") " Oct 30 19:23:43.393803 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.389313 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770e23c1-db57-4590-a46d-77ff7f9909ea-kube-api-access-6xzfz\" (UniqueName: \"kubernetes.io/projected/770e23c1-db57-4590-a46d-77ff7f9909ea-kube-api-access-6xzfz\") pod \"770e23c1-db57-4590-a46d-77ff7f9909ea\" (UID: \"770e23c1-db57-4590-a46d-77ff7f9909ea\") " Oct 30 19:23:43.393803 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.389393 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/770e23c1-db57-4590-a46d-77ff7f9909ea-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/770e23c1-db57-4590-a46d-77ff7f9909ea-archives-path\") pod \"770e23c1-db57-4590-a46d-77ff7f9909ea\" (UID: \"770e23c1-db57-4590-a46d-77ff7f9909ea\") " Oct 30 19:23:43.428041 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.421124 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770e23c1-db57-4590-a46d-77ff7f9909ea-archives-path" pod "770e23c1-db57-4590-a46d-77ff7f9909ea" (UID: "770e23c1-db57-4590-a46d-77ff7f9909ea"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:23:43.467169 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.447951 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" Oct 30 19:23:43.488037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.454891 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4sdcp-kxxdm" event={"ID":"770e23c1-db57-4590-a46d-77ff7f9909ea","Type":"ContainerDied","Data":"00c766d6b61e48285fe0bb0233de710bc15e541478ab5cc861f289bcffe8e2a6"} Oct 30 19:23:43.488516 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.488488 5047 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00c766d6b61e48285fe0bb0233de710bc15e541478ab5cc861f289bcffe8e2a6" Oct 30 19:23:43.494036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.459259 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770e23c1-db57-4590-a46d-77ff7f9909ea-service-ca-bundle" pod "770e23c1-db57-4590-a46d-77ff7f9909ea" (UID: "770e23c1-db57-4590-a46d-77ff7f9909ea"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:23:43.494261 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.471983 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770e23c1-db57-4590-a46d-77ff7f9909ea-kube-api-access-6xzfz" pod "770e23c1-db57-4590-a46d-77ff7f9909ea" (UID: "770e23c1-db57-4590-a46d-77ff7f9909ea"). InnerVolumeSpecName "kube-api-access-6xzfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:23:43.494377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.482271 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770e23c1-db57-4590-a46d-77ff7f9909ea-trusted-ca-bundle" pod "770e23c1-db57-4590-a46d-77ff7f9909ea" (UID: "770e23c1-db57-4590-a46d-77ff7f9909ea"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:23:43.529031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.505811 5047 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/770e23c1-db57-4590-a46d-77ff7f9909ea-trusted-ca-bundle\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:23:43.529281 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.529253 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xzfz\" (UniqueName: \"kubernetes.io/projected/770e23c1-db57-4590-a46d-77ff7f9909ea-kube-api-access-6xzfz\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:23:43.529432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.529418 5047 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/770e23c1-db57-4590-a46d-77ff7f9909ea-archives-path\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:23:43.529528 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.529517 5047 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/770e23c1-db57-4590-a46d-77ff7f9909ea-service-ca-bundle\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:23:43.571891 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:43.571779 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 30 19:23:44.340487 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:44.340444 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:23:44.340801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:44.340766 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" podUID="709e2b00-e43d-425d-855f-8b498883c049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:23:45.345855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:45.345408 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:23:45.345855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:45.345473 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" podUID="709e2b00-e43d-425d-855f-8b498883c049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:23:46.345031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:46.342456 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:23:46.345031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:46.342521 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" podUID="709e2b00-e43d-425d-855f-8b498883c049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:23:47.339630 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:47.339581 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:23:47.340054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:47.339657 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" podUID="709e2b00-e43d-425d-855f-8b498883c049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:23:48.344172 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:48.344137 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:48.344560 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:48.344520 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:23:48.345473 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:23:48.345445 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:24:07.233205 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:07.233172 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:24:07.235470 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:07.235444 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:24:09.606182 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:24:09.606140 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:24:19.212666 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.212443 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8"] Oct 30 19:24:19.215559 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.215530 5047 cpu_manager.go:414] "containerMap: removing container" podUID="770e23c1-db57-4590-a46d-77ff7f9909ea" containerName="insights-gathering" Oct 30 19:24:19.215776 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.215755 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770e23c1-db57-4590-a46d-77ff7f9909ea" containerName="insights-gathering" Oct 30 19:24:19.216430 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.216413 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="770e23c1-db57-4590-a46d-77ff7f9909ea" containerName="insights-gathering" Oct 30 19:24:19.236028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.229186 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8"] Oct 30 19:24:19.236028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.229362 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.298031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.284371 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk"] Oct 30 19:24:19.298031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.295487 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk"] Oct 30 19:24:19.298031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.295598 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.298031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.296053 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/074a1938-2380-4eb4-adf4-a15e3162507f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.298031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.296125 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.298031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.296174 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/074a1938-2380-4eb4-adf4-a15e3162507f-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.298031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.296237 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/074a1938-2380-4eb4-adf4-a15e3162507f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.298031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.296273 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.298031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.296328 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.298031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.296374 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.298031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.296429 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.298031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.296471 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkjgw\" (UniqueName: \"kubernetes.io/projected/074a1938-2380-4eb4-adf4-a15e3162507f-kube-api-access-vkjgw\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.398175 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.398133 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.398927 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.398890 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwzz7\" (UniqueName: \"kubernetes.io/projected/5fcb783a-2210-41a3-8379-b2c417ac94d8-kube-api-access-vwzz7\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.399075 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.398961 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/074a1938-2380-4eb4-adf4-a15e3162507f-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.399075 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.399031 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.399075 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.399059 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.399264 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.399108 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.399264 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.399142 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/074a1938-2380-4eb4-adf4-a15e3162507f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.399264 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.399188 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.399483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.399264 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.399483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.399300 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcb783a-2210-41a3-8379-b2c417ac94d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.399483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.399343 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.399483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.399380 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.399483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.399421 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.399483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.399468 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.399788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.399521 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkjgw\" (UniqueName: \"kubernetes.io/projected/074a1938-2380-4eb4-adf4-a15e3162507f-kube-api-access-vkjgw\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.399788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.399562 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.399788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.399612 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/074a1938-2380-4eb4-adf4-a15e3162507f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.399788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.399678 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.399788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.398819 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.400344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.400316 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.400647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.400625 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.409050 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.408972 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/074a1938-2380-4eb4-adf4-a15e3162507f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.409711 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.409685 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.418051 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.418024 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.420324 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.420274 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/074a1938-2380-4eb4-adf4-a15e3162507f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.434904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.434855 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/074a1938-2380-4eb4-adf4-a15e3162507f-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.435982 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.435923 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkjgw\" (UniqueName: \"kubernetes.io/projected/074a1938-2380-4eb4-adf4-a15e3162507f-kube-api-access-vkjgw\") pod \"maas-default-gateway-openshift-default-6f54779779-s2nd8\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.501158 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.501056 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.501158 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.501096 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.501158 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.501123 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.501467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.501181 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcb783a-2210-41a3-8379-b2c417ac94d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.501467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.501209 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.501467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.501236 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.501467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.501290 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.501676 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.501507 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.501676 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.501592 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwzz7\" (UniqueName: \"kubernetes.io/projected/5fcb783a-2210-41a3-8379-b2c417ac94d8-kube-api-access-vwzz7\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.501785 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.501665 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.501785 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.501714 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.501785 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.501763 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.501922 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.501800 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.502327 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.502300 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcb783a-2210-41a3-8379-b2c417ac94d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.503674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.503654 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.503885 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.503868 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.520626 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.520587 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.520743 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.520709 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwzz7\" (UniqueName: \"kubernetes.io/projected/5fcb783a-2210-41a3-8379-b2c417ac94d8-kube-api-access-vwzz7\") pod \"maas-default-gateway-openshift-default-6f54779779-frwdk\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.596047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.595999 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:19.619259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.619224 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:19.956345 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.956310 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8"] Oct 30 19:24:19.957693 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:24:19.957664 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074a1938_2380_4eb4_adf4_a15e3162507f.slice/crio-a0bf2528eb718ef2826371c88266e1ffb85cc418793a9260c25c6a16b25ba1a1 WatchSource:0}: Error finding container a0bf2528eb718ef2826371c88266e1ffb85cc418793a9260c25c6a16b25ba1a1: Status 404 returned error can't find the container with id a0bf2528eb718ef2826371c88266e1ffb85cc418793a9260c25c6a16b25ba1a1 Oct 30 19:24:19.960479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.960448 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:24:19.960567 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.960518 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:24:19.960567 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.960557 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:24:19.974514 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.974483 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" event={"ID":"074a1938-2380-4eb4-adf4-a15e3162507f","Type":"ContainerStarted","Data":"a0bf2528eb718ef2826371c88266e1ffb85cc418793a9260c25c6a16b25ba1a1"} Oct 30 19:24:19.992763 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:19.992734 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk"] Oct 30 19:24:20.002444 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:24:20.002414 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fcb783a_2210_41a3_8379_b2c417ac94d8.slice/crio-c65f8e7be2d504dd4a801fbbed3b605c66286a7930176f8c7534c3918cf59af0 WatchSource:0}: Error finding container c65f8e7be2d504dd4a801fbbed3b605c66286a7930176f8c7534c3918cf59af0: Status 404 returned error can't find the container with id c65f8e7be2d504dd4a801fbbed3b605c66286a7930176f8c7534c3918cf59af0 Oct 30 19:24:20.005088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:20.005054 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:24:20.005173 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:20.005123 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:24:20.005173 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:20.005154 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:24:20.992504 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:20.992470 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" event={"ID":"074a1938-2380-4eb4-adf4-a15e3162507f","Type":"ContainerStarted","Data":"1205eab80be08fefc58e3adf333e00b60a3985a3e3592125529b10fe242640b4"} Oct 30 19:24:20.994245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:20.994213 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" event={"ID":"5fcb783a-2210-41a3-8379-b2c417ac94d8","Type":"ContainerStarted","Data":"8f20a68f61c38434afa64fb51ff4b7758d2f38f6f132f02915d971141f61a0dd"} Oct 30 19:24:20.994375 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:20.994276 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" event={"ID":"5fcb783a-2210-41a3-8379-b2c417ac94d8","Type":"ContainerStarted","Data":"c65f8e7be2d504dd4a801fbbed3b605c66286a7930176f8c7534c3918cf59af0"} Oct 30 19:24:21.098087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:21.068538 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" podStartSLOduration=3.068517399 podStartE2EDuration="3.068517399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:21.054420735 +0000 UTC m=+1214.128876095" watchObservedRunningTime="2026-10-30 19:24:21.068517399 +0000 UTC m=+1214.142972761" Oct 30 19:24:21.129426 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:21.129360 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" podStartSLOduration=3.129339941 podStartE2EDuration="3.129339941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:21.125778682 +0000 UTC m=+1214.200234040" watchObservedRunningTime="2026-10-30 19:24:21.129339941 +0000 UTC m=+1214.203795303" Oct 30 19:24:21.596447 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:21.596416 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:21.597513 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:21.597484 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:24:21.597663 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:21.597613 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" podUID="074a1938-2380-4eb4-adf4-a15e3162507f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 19:24:21.647154 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:21.635562 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:21.647154 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:21.646853 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-frwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:24:21.668512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:21.668342 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" podUID="5fcb783a-2210-41a3-8379-b2c417ac94d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:24:22.597435 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:22.597386 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:24:22.597817 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:22.597464 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" podUID="074a1938-2380-4eb4-adf4-a15e3162507f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 19:24:22.620258 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:22.620219 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-frwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:24:22.620423 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:22.620287 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" podUID="5fcb783a-2210-41a3-8379-b2c417ac94d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:24:23.597231 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:23.597184 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:24:23.597424 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:23.597274 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" podUID="074a1938-2380-4eb4-adf4-a15e3162507f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 19:24:23.620665 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:23.620621 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-frwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:24:23.621114 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:23.620677 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" podUID="5fcb783a-2210-41a3-8379-b2c417ac94d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:24:24.597069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:24.597035 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:24:24.597238 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:24.597094 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" podUID="074a1938-2380-4eb4-adf4-a15e3162507f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 19:24:24.619936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:24.619899 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-frwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:24:24.620141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:24.619955 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" podUID="5fcb783a-2210-41a3-8379-b2c417ac94d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:24:25.191651 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.191357 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r"] Oct 30 19:24:25.197296 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.197272 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.215354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.215319 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r"] Oct 30 19:24:25.329822 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.329772 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.330823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.330174 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.330823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.330244 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58531637-258f-49e4-996d-088591db67e6-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.330823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.330280 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.330823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.330303 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58531637-258f-49e4-996d-088591db67e6-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.330823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.330352 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.330823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.330396 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.330823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.330420 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58531637-258f-49e4-996d-088591db67e6-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.330823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.330534 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r6c4\" (UniqueName: \"kubernetes.io/projected/58531637-258f-49e4-996d-088591db67e6-kube-api-access-5r6c4\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.432471 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.432427 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.432471 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.432472 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58531637-258f-49e4-996d-088591db67e6-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.432702 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.432539 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r6c4\" (UniqueName: \"kubernetes.io/projected/58531637-258f-49e4-996d-088591db67e6-kube-api-access-5r6c4\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.432702 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.432627 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.432702 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.432657 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.432702 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.432685 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58531637-258f-49e4-996d-088591db67e6-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.432828 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.432704 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.432828 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.432719 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58531637-258f-49e4-996d-088591db67e6-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.432828 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.432749 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.433556 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.433523 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.433704 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.433679 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.433813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.433798 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.433930 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.433908 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.455088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.434230 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58531637-258f-49e4-996d-088591db67e6-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.477031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.476124 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58531637-258f-49e4-996d-088591db67e6-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.477808 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.477771 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58531637-258f-49e4-996d-088591db67e6-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.491060 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.489475 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.491060 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.489774 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r6c4\" (UniqueName: \"kubernetes.io/projected/58531637-258f-49e4-996d-088591db67e6-kube-api-access-5r6c4\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.519596 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.519554 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:25.596886 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.596845 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:24:25.597134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.596896 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" podUID="074a1938-2380-4eb4-adf4-a15e3162507f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 19:24:25.620552 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.620450 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-frwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:24:25.620552 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.620514 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" podUID="5fcb783a-2210-41a3-8379-b2c417ac94d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:24:25.916040 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.915872 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r"] Oct 30 19:24:25.917650 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:24:25.917623 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58531637_258f_49e4_996d_088591db67e6.slice/crio-a9edf2f3a59a4430f66e68865cf51c4b32de426a7afa781ed7b0d21ae15f4beb WatchSource:0}: Error finding container a9edf2f3a59a4430f66e68865cf51c4b32de426a7afa781ed7b0d21ae15f4beb: Status 404 returned error can't find the container with id a9edf2f3a59a4430f66e68865cf51c4b32de426a7afa781ed7b0d21ae15f4beb Oct 30 19:24:25.920575 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.920532 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:24:25.920677 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.920626 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:24:25.920733 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:25.920673 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:24:26.028893 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:26.028858 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" event={"ID":"58531637-258f-49e4-996d-088591db67e6","Type":"ContainerStarted","Data":"a9edf2f3a59a4430f66e68865cf51c4b32de426a7afa781ed7b0d21ae15f4beb"} Oct 30 19:24:26.596745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:26.596619 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:24:26.596745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:26.596680 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" podUID="074a1938-2380-4eb4-adf4-a15e3162507f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 19:24:26.620701 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:26.620665 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-frwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:24:26.620873 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:26.620730 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" podUID="5fcb783a-2210-41a3-8379-b2c417ac94d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:24:27.087318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:27.087260 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" event={"ID":"58531637-258f-49e4-996d-088591db67e6","Type":"ContainerStarted","Data":"b94f63b780a15ae9610be20f8fa3d4ddedd5c4d7d2f27a2efdfaf2701cd020e8"} Oct 30 19:24:27.196035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:27.195642 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" podStartSLOduration=2.195622288 podStartE2EDuration="2.195622288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:27.148669291 +0000 UTC m=+1220.223124651" watchObservedRunningTime="2026-10-30 19:24:27.195622288 +0000 UTC m=+1220.270077649" Oct 30 19:24:27.520098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:27.519963 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:27.522134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:27.522102 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:24:27.522308 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:27.522161 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" podUID="58531637-258f-49e4-996d-088591db67e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:24:27.596929 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:27.596877 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:24:27.597381 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:27.596940 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" podUID="074a1938-2380-4eb4-adf4-a15e3162507f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 19:24:27.620269 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:27.620221 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-frwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:24:27.620424 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:27.620307 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" podUID="5fcb783a-2210-41a3-8379-b2c417ac94d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:24:28.523977 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:28.523939 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:24:28.524310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:28.524276 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" podUID="58531637-258f-49e4-996d-088591db67e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:24:29.521348 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:29.521296 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:24:29.521789 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:29.521377 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" podUID="58531637-258f-49e4-996d-088591db67e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:24:29.587533 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:29.587496 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:29.588521 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:29.588491 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:29.590926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:29.590892 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 19:24:29.597830 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:29.597790 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:24:29.597989 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:29.597845 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" podUID="074a1938-2380-4eb4-adf4-a15e3162507f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:24:29.611592 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:29.610779 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:29.612295 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:29.612238 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:29.615340 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:29.615271 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:24:30.520340 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:30.520303 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:24:30.520516 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:30.520385 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" podUID="58531637-258f-49e4-996d-088591db67e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:24:31.521091 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:31.521059 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:24:31.521568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:31.521536 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" podUID="58531637-258f-49e4-996d-088591db67e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:24:32.520547 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:32.520496 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:24:32.520867 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:32.520830 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" podUID="58531637-258f-49e4-996d-088591db67e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:24:34.110783 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:34.110750 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:34.111300 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:34.111203 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:34.114723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:34.112631 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:24:39.601783 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:24:39.601742 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:24:45.213403 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.213368 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5"] Oct 30 19:24:45.218818 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.218785 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.249688 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.249653 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5"] Oct 30 19:24:45.326405 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.326366 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.326559 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.326422 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.326559 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.326528 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7m96\" (UniqueName: \"kubernetes.io/projected/586ca704-ecaf-4d51-88b4-6ea6770bb10f-kube-api-access-g7m96\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.326659 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.326573 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.326701 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.326688 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.326736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.326712 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.326736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.326732 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.326836 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.326821 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.326896 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.326882 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.427576 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.427544 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.427768 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.427595 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.427768 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.427637 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7m96\" (UniqueName: \"kubernetes.io/projected/586ca704-ecaf-4d51-88b4-6ea6770bb10f-kube-api-access-g7m96\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.427768 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.427672 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.427768 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.427728 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.427768 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.427747 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.427768 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.427767 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.428333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.427826 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.428333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.427898 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.428333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.428177 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.428524 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.428498 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.428613 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.428539 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.428751 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.428727 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.428822 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.428738 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.430730 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.430701 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.430829 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.430786 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.446882 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.446857 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.447267 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.447179 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7m96\" (UniqueName: \"kubernetes.io/projected/586ca704-ecaf-4d51-88b4-6ea6770bb10f-kube-api-access-g7m96\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.533436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.533340 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:45.788368 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.788277 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj"] Oct 30 19:24:45.794039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.793998 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.814977 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.814946 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj"] Oct 30 19:24:45.832815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.832782 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.832976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.832818 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw7ld\" (UniqueName: \"kubernetes.io/projected/9399ee4d-b327-4739-85b1-d18eecea8116-kube-api-access-fw7ld\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.832976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.832850 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9399ee4d-b327-4739-85b1-d18eecea8116-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.833098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.833047 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.833098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.833079 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.833180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.833112 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.833215 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.833204 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.833299 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.833255 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9399ee4d-b327-4739-85b1-d18eecea8116-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.833394 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.833364 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9399ee4d-b327-4739-85b1-d18eecea8116-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.862635 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.862602 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5"] Oct 30 19:24:45.863611 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:24:45.863583 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586ca704_ecaf_4d51_88b4_6ea6770bb10f.slice/crio-d52759bf76bc9e6ac70849c373248bb4d73cb23fffa6aeab94874534e39853f2 WatchSource:0}: Error finding container d52759bf76bc9e6ac70849c373248bb4d73cb23fffa6aeab94874534e39853f2: Status 404 returned error can't find the container with id d52759bf76bc9e6ac70849c373248bb4d73cb23fffa6aeab94874534e39853f2 Oct 30 19:24:45.866226 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.866196 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:24:45.866300 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.866260 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:24:45.866300 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.866292 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:24:45.934508 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.934475 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9399ee4d-b327-4739-85b1-d18eecea8116-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.934619 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.934539 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9399ee4d-b327-4739-85b1-d18eecea8116-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.934680 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.934630 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.934826 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.934800 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw7ld\" (UniqueName: \"kubernetes.io/projected/9399ee4d-b327-4739-85b1-d18eecea8116-kube-api-access-fw7ld\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.934908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.934882 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9399ee4d-b327-4739-85b1-d18eecea8116-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.934962 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.934950 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.935090 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.934974 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.935090 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.934999 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.935200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.935088 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.935200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.935159 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.935302 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.935283 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.935412 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.935383 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.935516 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.935493 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9399ee4d-b327-4739-85b1-d18eecea8116-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.935583 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.935519 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.937391 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.937366 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9399ee4d-b327-4739-85b1-d18eecea8116-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.937503 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.937427 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.958874 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.958848 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9399ee4d-b327-4739-85b1-d18eecea8116-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:45.959350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:45.959325 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw7ld\" (UniqueName: \"kubernetes.io/projected/9399ee4d-b327-4739-85b1-d18eecea8116-kube-api-access-fw7ld\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:46.117032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:46.116360 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:46.438682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:46.438585 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" event={"ID":"586ca704-ecaf-4d51-88b4-6ea6770bb10f","Type":"ContainerStarted","Data":"e47cd3dfe05e82d988c03f5cecafe8c7b4ff400a2c4883f34dbd29dad24639b3"} Oct 30 19:24:46.438682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:46.438674 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" event={"ID":"586ca704-ecaf-4d51-88b4-6ea6770bb10f","Type":"ContainerStarted","Data":"d52759bf76bc9e6ac70849c373248bb4d73cb23fffa6aeab94874534e39853f2"} Oct 30 19:24:46.454203 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:46.454171 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj"] Oct 30 19:24:46.461189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:24:46.461128 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9399ee4d_b327_4739_85b1_d18eecea8116.slice/crio-23b398febb3b415ce230df545bd77c8fd3e00dbbff38cef5ccc657999be64bee WatchSource:0}: Error finding container 23b398febb3b415ce230df545bd77c8fd3e00dbbff38cef5ccc657999be64bee: Status 404 returned error can't find the container with id 23b398febb3b415ce230df545bd77c8fd3e00dbbff38cef5ccc657999be64bee Oct 30 19:24:46.464390 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:46.464346 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:24:46.464640 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:46.464608 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:24:46.464756 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:46.464688 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:24:46.534593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:46.534557 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:46.536165 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:46.536137 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:24:46.536249 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:46.536187 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" podUID="586ca704-ecaf-4d51-88b4-6ea6770bb10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:24:47.487669 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:47.487624 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" event={"ID":"9399ee4d-b327-4739-85b1-d18eecea8116","Type":"ContainerStarted","Data":"5437db989a834dc6349d6ce5a9c5cf7ba10752132fa498067855c29f51b0380b"} Oct 30 19:24:47.488146 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:47.487691 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" event={"ID":"9399ee4d-b327-4739-85b1-d18eecea8116","Type":"ContainerStarted","Data":"23b398febb3b415ce230df545bd77c8fd3e00dbbff38cef5ccc657999be64bee"} Oct 30 19:24:47.560038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:47.558415 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:24:47.560038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:47.558480 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" podUID="586ca704-ecaf-4d51-88b4-6ea6770bb10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:24:47.560038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:47.558480 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" podStartSLOduration=2.558456919 podStartE2EDuration="2.558456919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:46.489947864 +0000 UTC m=+1239.564403222" watchObservedRunningTime="2026-10-30 19:24:47.558456919 +0000 UTC m=+1240.632912279" Oct 30 19:24:47.583031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:47.578440 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" podStartSLOduration=2.5784167030000003 podStartE2EDuration="2.578416703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:47.528624662 +0000 UTC m=+1240.603080020" watchObservedRunningTime="2026-10-30 19:24:47.578416703 +0000 UTC m=+1240.652872063" Oct 30 19:24:48.120031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:48.119372 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:48.124029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:48.121756 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:24:48.124029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:48.121836 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" podUID="9399ee4d-b327-4739-85b1-d18eecea8116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 19:24:48.538089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:48.535128 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:24:48.538089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:48.535195 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" podUID="586ca704-ecaf-4d51-88b4-6ea6770bb10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:24:49.117717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:49.117668 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:24:49.117926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:49.117746 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" podUID="9399ee4d-b327-4739-85b1-d18eecea8116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 19:24:49.538088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:49.533802 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:24:49.538088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:49.533859 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" podUID="586ca704-ecaf-4d51-88b4-6ea6770bb10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:24:50.117673 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:50.117634 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:24:50.117964 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:50.117727 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" podUID="9399ee4d-b327-4739-85b1-d18eecea8116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 19:24:50.559087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:50.558431 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:24:50.559087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:50.558516 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" podUID="586ca704-ecaf-4d51-88b4-6ea6770bb10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:24:51.117386 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:51.117334 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:24:51.117574 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:51.117407 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" podUID="9399ee4d-b327-4739-85b1-d18eecea8116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 19:24:51.555088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:51.538291 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:24:51.555088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:51.543045 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" podUID="586ca704-ecaf-4d51-88b4-6ea6770bb10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:24:52.117077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:52.117036 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:24:52.117489 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:52.117109 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" podUID="9399ee4d-b327-4739-85b1-d18eecea8116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 19:24:52.545088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:52.538405 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:24:52.545088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:52.538484 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" podUID="586ca704-ecaf-4d51-88b4-6ea6770bb10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:24:53.117998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:53.117959 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:24:53.118459 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:53.118050 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" podUID="9399ee4d-b327-4739-85b1-d18eecea8116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 19:24:53.568591 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:53.568489 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:53.570105 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:53.569923 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:53.587025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:53.585279 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:24:54.121581 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:54.121547 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:54.122031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:54.121998 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:54.122816 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:54.122794 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:24:57.105305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.105271 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6"] Oct 30 19:24:57.109897 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.109866 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.130068 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.130033 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6"] Oct 30 19:24:57.134937 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.134902 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e67b84\"/\"openshift-service-ca.crt\"" Oct 30 19:24:57.134937 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.134915 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e67b84\"/\"distinct-w5-e67b84-kserve-self-signed-certs\"" Oct 30 19:24:57.136298 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.136257 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e67b84\"/\"default-dockercfg-q2bsm\"" Oct 30 19:24:57.136298 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.136289 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e67b84\"/\"kube-root-ca.crt\"" Oct 30 19:24:57.289525 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.289486 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcz5q\" (UniqueName: \"kubernetes.io/projected/55ce8a09-3219-4883-adf2-452d0dd05b18-kube-api-access-gcz5q\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.289703 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.289557 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-model-cache\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.289752 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.289722 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ce8a09-3219-4883-adf2-452d0dd05b18-tls-certs\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.289793 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.289777 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-tmp-dir\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.289841 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.289828 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-home\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.289913 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.289901 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-dshm\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.409597 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.398967 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcz5q\" (UniqueName: \"kubernetes.io/projected/55ce8a09-3219-4883-adf2-452d0dd05b18-kube-api-access-gcz5q\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.409597 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.399123 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-model-cache\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.409597 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.399222 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ce8a09-3219-4883-adf2-452d0dd05b18-tls-certs\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.409597 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.399280 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-tmp-dir\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.409597 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.399316 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-home\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.409597 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.399375 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-dshm\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.411220 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.411173 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-tmp-dir\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.412084 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.412046 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-model-cache\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.412598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.412561 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-home\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.462950 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.462919 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-dshm\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.475704 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.475670 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ce8a09-3219-4883-adf2-452d0dd05b18-tls-certs\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.479817 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.479710 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcz5q\" (UniqueName: \"kubernetes.io/projected/55ce8a09-3219-4883-adf2-452d0dd05b18-kube-api-access-gcz5q\") pod \"distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:57.728352 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:57.728269 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:58.031146 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:58.031114 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6"] Oct 30 19:24:58.032374 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:24:58.032340 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55ce8a09_3219_4883_adf2_452d0dd05b18.slice/crio-06129f88a45de52bfa8adfa5d3e6895e15f7125c8a93c2f69e1215d1ae2fd58a WatchSource:0}: Error finding container 06129f88a45de52bfa8adfa5d3e6895e15f7125c8a93c2f69e1215d1ae2fd58a: Status 404 returned error can't find the container with id 06129f88a45de52bfa8adfa5d3e6895e15f7125c8a93c2f69e1215d1ae2fd58a Oct 30 19:24:58.687027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:58.686828 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" event={"ID":"55ce8a09-3219-4883-adf2-452d0dd05b18","Type":"ContainerStarted","Data":"3c5a3b406ee4b4e127256a19241ec5d9a97266a91c40e1e2b9aafc37a3689c6b"} Oct 30 19:24:58.687027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:58.686881 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" event={"ID":"55ce8a09-3219-4883-adf2-452d0dd05b18","Type":"ContainerStarted","Data":"06129f88a45de52bfa8adfa5d3e6895e15f7125c8a93c2f69e1215d1ae2fd58a"} Oct 30 19:24:58.728806 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:58.728767 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:24:58.729521 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:24:58.729122 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podStartSLOduration=1.729101228 podStartE2EDuration="1.729101228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:58.725683859 +0000 UTC m=+1251.800139215" watchObservedRunningTime="2026-10-30 19:24:58.729101228 +0000 UTC m=+1251.803556599" Oct 30 19:25:07.728627 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:07.728576 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:25:07.741684 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:07.741652 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:25:07.760896 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:07.760858 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:25:09.718214 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:25:09.718162 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:25:30.356449 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.356398 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg"] Oct 30 19:25:30.361936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.361903 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.395048 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.388290 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg"] Oct 30 19:25:30.399426 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.398834 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.399964 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.399934 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.520099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.514455 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h68z\" (UniqueName: \"kubernetes.io/projected/bccbf42c-7950-485f-a5b4-0475f827c419-kube-api-access-2h68z\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.520099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.514539 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bccbf42c-7950-485f-a5b4-0475f827c419-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.520099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.514654 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.520099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.514685 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bccbf42c-7950-485f-a5b4-0475f827c419-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.520099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.514746 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.520099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.514859 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.520099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.514888 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.520099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.514912 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.520099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.514964 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bccbf42c-7950-485f-a5b4-0475f827c419-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.520099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.515582 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.520099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.515860 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.553707 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.537864 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt"] Oct 30 19:25:30.569293 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.567039 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.569293 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.568773 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt"] Oct 30 19:25:30.616736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.616643 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.616736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.616681 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bccbf42c-7950-485f-a5b4-0475f827c419-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.616736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.616709 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwd65\" (UniqueName: \"kubernetes.io/projected/98e541fb-4108-4b0a-8c6b-7e6587354f19-kube-api-access-fwd65\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.616989 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.616774 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.616989 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.616900 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.616989 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.616930 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e541fb-4108-4b0a-8c6b-7e6587354f19-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.617197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.617113 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.617197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.617152 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.617302 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.617216 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bccbf42c-7950-485f-a5b4-0475f827c419-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.617405 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.617385 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.617466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.617446 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.617502 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.617491 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.617550 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.617523 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h68z\" (UniqueName: \"kubernetes.io/projected/bccbf42c-7950-485f-a5b4-0475f827c419-kube-api-access-2h68z\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.617613 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.617589 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.617701 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.617680 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bccbf42c-7950-485f-a5b4-0475f827c419-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.617760 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.617717 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.617760 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.617750 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.617863 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.617808 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.617951 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.617927 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bccbf42c-7950-485f-a5b4-0475f827c419-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.619946 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.619916 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.620392 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.620371 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bccbf42c-7950-485f-a5b4-0475f827c419-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.637583 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.637546 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bccbf42c-7950-485f-a5b4-0475f827c419-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.637730 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.637561 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h68z\" (UniqueName: \"kubernetes.io/projected/bccbf42c-7950-485f-a5b4-0475f827c419-kube-api-access-2h68z\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.695349 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.695310 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:30.718711 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.718682 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e541fb-4108-4b0a-8c6b-7e6587354f19-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.718895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.718827 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.718895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.718855 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.718895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.718891 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.719097 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.719069 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.719154 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.719134 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.719212 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.719188 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.719311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.719292 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwd65\" (UniqueName: \"kubernetes.io/projected/98e541fb-4108-4b0a-8c6b-7e6587354f19-kube-api-access-fwd65\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.719381 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.719307 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.719381 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.719369 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.719381 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.719369 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.719642 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.719619 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e541fb-4108-4b0a-8c6b-7e6587354f19-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.719806 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.719786 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.719865 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.719818 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.722075 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.722050 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.722227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.722205 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.737729 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.737692 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.737882 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.737761 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwd65\" (UniqueName: \"kubernetes.io/projected/98e541fb-4108-4b0a-8c6b-7e6587354f19-kube-api-access-fwd65\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:30.896357 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:30.896266 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:31.024621 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:31.024593 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg"] Oct 30 19:25:31.025763 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:25:31.025733 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbccbf42c_7950_485f_a5b4_0475f827c419.slice/crio-9efd9cbdc5f30109e725d1103e4b2b00f404cea8a7b80759b19ef2bb36ddc98a WatchSource:0}: Error finding container 9efd9cbdc5f30109e725d1103e4b2b00f404cea8a7b80759b19ef2bb36ddc98a: Status 404 returned error can't find the container with id 9efd9cbdc5f30109e725d1103e4b2b00f404cea8a7b80759b19ef2bb36ddc98a Oct 30 19:25:31.029077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:31.029037 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:25:31.029169 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:31.029133 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:25:31.029207 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:31.029181 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:25:31.051477 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:31.051450 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" event={"ID":"bccbf42c-7950-485f-a5b4-0475f827c419","Type":"ContainerStarted","Data":"9efd9cbdc5f30109e725d1103e4b2b00f404cea8a7b80759b19ef2bb36ddc98a"} Oct 30 19:25:31.205820 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:31.205784 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt"] Oct 30 19:25:31.206190 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:25:31.206167 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e541fb_4108_4b0a_8c6b_7e6587354f19.slice/crio-d8777e08d1888c054ea87d132ca3012357bd1783a54c2c38aebe5cf9e569b230 WatchSource:0}: Error finding container d8777e08d1888c054ea87d132ca3012357bd1783a54c2c38aebe5cf9e569b230: Status 404 returned error can't find the container with id d8777e08d1888c054ea87d132ca3012357bd1783a54c2c38aebe5cf9e569b230 Oct 30 19:25:31.209697 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:31.209663 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:25:31.209788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:31.209730 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:25:31.209788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:31.209760 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:25:32.098141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:32.098086 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" event={"ID":"98e541fb-4108-4b0a-8c6b-7e6587354f19","Type":"ContainerStarted","Data":"65c4ccb3168fb3d14dc1a87ad072693769ee49757b1868aa2777d51bdb1517c3"} Oct 30 19:25:32.098770 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:32.098730 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" event={"ID":"98e541fb-4108-4b0a-8c6b-7e6587354f19","Type":"ContainerStarted","Data":"d8777e08d1888c054ea87d132ca3012357bd1783a54c2c38aebe5cf9e569b230"} Oct 30 19:25:32.101551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:32.101524 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" event={"ID":"bccbf42c-7950-485f-a5b4-0475f827c419","Type":"ContainerStarted","Data":"be8cd1ecb2db0cc524b9c5aad84833f17c7d32edd9222b650338660bf689cb4c"} Oct 30 19:25:32.207383 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:32.207325 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" podStartSLOduration=2.207308816 podStartE2EDuration="2.207308816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:32.155544946 +0000 UTC m=+1285.230000307" watchObservedRunningTime="2026-10-30 19:25:32.207308816 +0000 UTC m=+1285.281764174" Oct 30 19:25:32.208633 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:32.208599 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" podStartSLOduration=2.208584491 podStartE2EDuration="2.208584491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:32.20682494 +0000 UTC m=+1285.281280298" watchObservedRunningTime="2026-10-30 19:25:32.208584491 +0000 UTC m=+1285.283039850" Oct 30 19:25:32.695771 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:32.695647 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:32.701028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:32.699392 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:25:32.701028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:32.699453 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" podUID="bccbf42c-7950-485f-a5b4-0475f827c419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 19:25:32.896784 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:32.896740 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:32.898130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:32.898109 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:25:32.898288 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:32.898266 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" podUID="98e541fb-4108-4b0a-8c6b-7e6587354f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 19:25:33.695963 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:33.695916 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:25:33.696361 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:33.695980 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" podUID="bccbf42c-7950-485f-a5b4-0475f827c419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 19:25:33.853369 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:33.853334 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4"] Oct 30 19:25:33.900266 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:33.900220 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:25:33.900465 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:33.900295 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" podUID="98e541fb-4108-4b0a-8c6b-7e6587354f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 19:25:33.901584 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:33.901556 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4"] Oct 30 19:25:33.901766 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:33.901692 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:33.929764 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:33.929727 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e67b84\"/\"distinct-2-w5-e67b84-kserve-self-signed-certs\"" Oct 30 19:25:33.997624 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:33.997535 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mchkj\" (UniqueName: \"kubernetes.io/projected/d6339b71-2746-47d8-810b-217e48b23e0e-kube-api-access-mchkj\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:33.997827 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:33.997812 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-home\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:33.997908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:33.997896 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-tmp-dir\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:33.998071 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:33.998052 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-dshm\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:33.998233 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:33.998217 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-model-cache\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:33.998405 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:33.998393 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6339b71-2746-47d8-810b-217e48b23e0e-tls-certs\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:34.132286 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:34.131920 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6339b71-2746-47d8-810b-217e48b23e0e-tls-certs\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:34.132286 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:34.132104 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mchkj\" (UniqueName: \"kubernetes.io/projected/d6339b71-2746-47d8-810b-217e48b23e0e-kube-api-access-mchkj\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:34.132286 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:34.132148 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-home\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:34.132286 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:34.132173 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-tmp-dir\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:34.132286 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:34.132216 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-dshm\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:34.132286 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:34.132270 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-model-cache\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:34.132858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:34.132831 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-model-cache\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:34.133146 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:34.133122 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-home\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:34.156124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:34.156087 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6339b71-2746-47d8-810b-217e48b23e0e-tls-certs\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:34.181932 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:34.181818 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mchkj\" (UniqueName: \"kubernetes.io/projected/d6339b71-2746-47d8-810b-217e48b23e0e-kube-api-access-mchkj\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:34.195598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:34.182358 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-tmp-dir\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:34.222042 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:34.221578 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-dshm\") pod \"distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:34.521681 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:34.521634 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:34.696937 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:34.696200 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:25:34.696937 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:34.696266 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" podUID="bccbf42c-7950-485f-a5b4-0475f827c419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 19:25:34.899045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:34.897332 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:25:34.899045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:34.897386 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" podUID="98e541fb-4108-4b0a-8c6b-7e6587354f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 19:25:35.132695 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:35.132664 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4"] Oct 30 19:25:35.133586 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:25:35.133556 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6339b71_2746_47d8_810b_217e48b23e0e.slice/crio-1a24d36b7fe5e635e5cc26fad29e066ee253ef5e61f95b9dec88b523844936ef WatchSource:0}: Error finding container 1a24d36b7fe5e635e5cc26fad29e066ee253ef5e61f95b9dec88b523844936ef: Status 404 returned error can't find the container with id 1a24d36b7fe5e635e5cc26fad29e066ee253ef5e61f95b9dec88b523844936ef Oct 30 19:25:35.205751 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:35.205718 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" event={"ID":"d6339b71-2746-47d8-810b-217e48b23e0e","Type":"ContainerStarted","Data":"1a24d36b7fe5e635e5cc26fad29e066ee253ef5e61f95b9dec88b523844936ef"} Oct 30 19:25:35.696449 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:35.696409 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:25:35.696649 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:35.696473 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" podUID="bccbf42c-7950-485f-a5b4-0475f827c419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 19:25:35.897431 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:35.897383 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:25:35.897880 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:35.897446 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" podUID="98e541fb-4108-4b0a-8c6b-7e6587354f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 19:25:36.214166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:36.214127 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" event={"ID":"d6339b71-2746-47d8-810b-217e48b23e0e","Type":"ContainerStarted","Data":"714a7ead6ec214e8ddc5ec5e619959eb3c8d4fb4caf8562196648361dc820b89"} Oct 30 19:25:36.525207 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:36.522303 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:36.695961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:36.695865 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:25:36.695961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:36.695928 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" podUID="bccbf42c-7950-485f-a5b4-0475f827c419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 19:25:36.900029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:36.899291 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:25:36.900029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:36.899379 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" podUID="98e541fb-4108-4b0a-8c6b-7e6587354f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 19:25:37.695961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:37.695915 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:25:37.696167 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:37.695990 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" podUID="bccbf42c-7950-485f-a5b4-0475f827c419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 19:25:37.897392 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:37.897351 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:25:37.897574 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:37.897415 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" podUID="98e541fb-4108-4b0a-8c6b-7e6587354f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 19:25:38.913641 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:38.913605 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:38.914096 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:38.914067 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:38.915172 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:38.915131 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 19:25:38.966451 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:38.966395 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podStartSLOduration=5.96637902 podStartE2EDuration="5.96637902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:36.256302966 +0000 UTC m=+1289.330758326" watchObservedRunningTime="2026-10-30 19:25:38.96637902 +0000 UTC m=+1292.040834378" Oct 30 19:25:39.003526 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:39.003490 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:39.003927 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:39.003868 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:39.005206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:39.005184 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:25:39.602116 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:25:39.602078 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:25:44.529278 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:44.529230 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:44.731427 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:44.731395 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:25:45.370103 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:25:45.370073 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:26:09.603256 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:26:09.603213 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:26:15.478278 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.478243 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95"] Oct 30 19:26:15.483983 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.483959 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.506839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.506805 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95"] Oct 30 19:26:15.592291 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.592247 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.592488 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.592303 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708f310c-157e-4c22-8de1-1bdecacc2023-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.592488 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.592393 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.592584 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.592561 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.592630 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.592593 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.592683 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.592634 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708f310c-157e-4c22-8de1-1bdecacc2023-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.592789 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.592773 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb5zk\" (UniqueName: \"kubernetes.io/projected/708f310c-157e-4c22-8de1-1bdecacc2023-kube-api-access-hb5zk\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.592864 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.592836 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708f310c-157e-4c22-8de1-1bdecacc2023-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.592993 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.592970 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.694317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.694278 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.694529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.694330 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.694529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.694356 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708f310c-157e-4c22-8de1-1bdecacc2023-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.694529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.694374 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.694529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.694474 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.694529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.694500 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.694812 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.694540 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708f310c-157e-4c22-8de1-1bdecacc2023-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.694812 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.694596 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb5zk\" (UniqueName: \"kubernetes.io/projected/708f310c-157e-4c22-8de1-1bdecacc2023-kube-api-access-hb5zk\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.694812 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.694666 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708f310c-157e-4c22-8de1-1bdecacc2023-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.694812 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.694794 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-credential-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.694957 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.694830 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-workload-certs\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.694957 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.694901 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-workload-socket\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.695095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.695071 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-istio-data\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.695322 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.695293 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708f310c-157e-4c22-8de1-1bdecacc2023-istiod-ca-cert\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.697037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.696997 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-istio-envoy\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.697309 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.697289 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708f310c-157e-4c22-8de1-1bdecacc2023-istio-podinfo\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.713218 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.713184 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb5zk\" (UniqueName: \"kubernetes.io/projected/708f310c-157e-4c22-8de1-1bdecacc2023-kube-api-access-hb5zk\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.713358 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.713300 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708f310c-157e-4c22-8de1-1bdecacc2023-istio-token\") pod \"e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:15.801137 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:15.801037 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:16.116216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:16.116186 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95"] Oct 30 19:26:16.116958 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:26:16.116931 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708f310c_157e_4c22_8de1_1bdecacc2023.slice/crio-20d3a3d7259b4bc0e2eba23d076670d742646ee54aba4a80fb3b214c29ccf3f4 WatchSource:0}: Error finding container 20d3a3d7259b4bc0e2eba23d076670d742646ee54aba4a80fb3b214c29ccf3f4: Status 404 returned error can't find the container with id 20d3a3d7259b4bc0e2eba23d076670d742646ee54aba4a80fb3b214c29ccf3f4 Oct 30 19:26:16.119606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:16.119576 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:26:16.119689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:16.119644 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:26:16.119689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:16.119675 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:26:16.757335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:16.757288 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" event={"ID":"708f310c-157e-4c22-8de1-1bdecacc2023","Type":"ContainerStarted","Data":"fec855be18ec33febbacea02dd182403570d82b24bbd4abaa97d9c1c7868e056"} Oct 30 19:26:16.757976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:16.757932 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" event={"ID":"708f310c-157e-4c22-8de1-1bdecacc2023","Type":"ContainerStarted","Data":"20d3a3d7259b4bc0e2eba23d076670d742646ee54aba4a80fb3b214c29ccf3f4"} Oct 30 19:26:16.802103 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:16.802067 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:16.803308 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:16.803264 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" podStartSLOduration=1.803249173 podStartE2EDuration="1.803249173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:16.799628931 +0000 UTC m=+1329.874084291" watchObservedRunningTime="2026-10-30 19:26:16.803249173 +0000 UTC m=+1329.877704531" Oct 30 19:26:17.804127 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:17.804095 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:26:17.804553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:17.804153 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" podUID="708f310c-157e-4c22-8de1-1bdecacc2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:26:18.801965 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:18.801926 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:26:18.802198 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:18.801984 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" podUID="708f310c-157e-4c22-8de1-1bdecacc2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:26:19.801786 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:19.801739 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:26:19.802312 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:19.801812 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" podUID="708f310c-157e-4c22-8de1-1bdecacc2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:26:20.801868 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:20.801829 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:26:20.802324 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:20.801909 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" podUID="708f310c-157e-4c22-8de1-1bdecacc2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:26:21.820500 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:21.820166 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:26:21.820500 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:21.820245 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" podUID="708f310c-157e-4c22-8de1-1bdecacc2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:26:22.801523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:22.801487 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:26:22.801731 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:22.801555 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" podUID="708f310c-157e-4c22-8de1-1bdecacc2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:26:23.807462 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:23.807430 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:23.807925 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:23.807763 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:23.808713 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:23.808693 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:26:25.607807 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.607769 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6"] Oct 30 19:26:25.631499 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.631456 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.662940 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.662915 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p"] Oct 30 19:26:25.678571 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.678527 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.679555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.678868 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.679555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.678918 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.679555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.679025 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfzgz\" (UniqueName: \"kubernetes.io/projected/97681db1-a8e8-4ad9-a97f-b914a8c55c02-kube-api-access-lfzgz\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.679555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.679059 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.679555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.679158 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.679555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.679212 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.679555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.679274 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.679555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.679353 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.698032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.686809 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:25.810130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.810091 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.829045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.810552 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.829045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.810604 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.829045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.810637 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.829045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.810718 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfzgz\" (UniqueName: \"kubernetes.io/projected/97681db1-a8e8-4ad9-a97f-b914a8c55c02-kube-api-access-lfzgz\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.829045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.810750 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.829045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.810842 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.829045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.810891 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.829045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.810935 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.829045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.811446 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.829045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.812177 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.829045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.812505 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.847100 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.847045 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.848486 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.848437 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:25.871074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.870950 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6"] Oct 30 19:26:25.928029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.926338 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:25.928029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.926417 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:25.928029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.926466 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6p77\" (UniqueName: \"kubernetes.io/projected/d2ceec35-f806-4418-aebd-f515b2423b4d-kube-api-access-q6p77\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:25.928029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.926502 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:25.928029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.926553 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2ceec35-f806-4418-aebd-f515b2423b4d-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:25.931251 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.931213 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:25.931742 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.931715 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:25.969272 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.969236 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:25.969730 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:25.969696 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.027744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.024628 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:26.095829 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.062837 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:26.097203 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.097162 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.097501 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.097478 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.097657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.097640 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.097805 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.097787 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6p77\" (UniqueName: \"kubernetes.io/projected/d2ceec35-f806-4418-aebd-f515b2423b4d-kube-api-access-q6p77\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.097977 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.097935 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.098169 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.098111 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2ceec35-f806-4418-aebd-f515b2423b4d-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.124215 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.124127 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-credential-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.124761 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.124741 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:26.126353 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.126333 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-workload-certs\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.149030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.126602 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.161146 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.161105 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.161423 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.161402 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.186671 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.127700 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2ceec35-f806-4418-aebd-f515b2423b4d-istiod-ca-cert\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.187195 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.181832 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-workload-socket\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.187957 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.187930 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfzgz\" (UniqueName: \"kubernetes.io/projected/97681db1-a8e8-4ad9-a97f-b914a8c55c02-kube-api-access-lfzgz\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:26.220030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.200774 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-data\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.250159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.249900 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-podinfo\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.266953 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.266877 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6p77\" (UniqueName: \"kubernetes.io/projected/d2ceec35-f806-4418-aebd-f515b2423b4d-kube-api-access-q6p77\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.303597 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.303291 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:26.312282 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.312240 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-token\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.312864 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.312834 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-envoy\") pod \"e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.355362 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.323399 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p"] Oct 30 19:26:26.380262 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.380167 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 19:26:26.404583 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.404251 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:26.619416 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:26.615307 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:27.404165 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:27.380604 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 19:26:27.536221 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:27.530070 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:26:27.536221 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:27.530166 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:26:28.175233 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:28.174139 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:26:28.175233 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:28.174251 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:26:28.175233 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:28.174299 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:26:28.183979 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:26:28.183940 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97681db1_a8e8_4ad9_a97f_b914a8c55c02.slice/crio-1ce29c3dfbd69c570c71c18496232db1cc5f5fcc08e48fc0284d0b98ca98daad WatchSource:0}: Error finding container 1ce29c3dfbd69c570c71c18496232db1cc5f5fcc08e48fc0284d0b98ca98daad: Status 404 returned error can't find the container with id 1ce29c3dfbd69c570c71c18496232db1cc5f5fcc08e48fc0284d0b98ca98daad Oct 30 19:26:28.190303 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:28.190058 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6"] Oct 30 19:26:28.266937 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:28.266898 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p"] Oct 30 19:26:28.288449 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:28.288405 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:26:28.288731 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:28.288707 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:26:28.288927 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:28.288908 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:26:28.406621 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:28.400840 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" event={"ID":"97681db1-a8e8-4ad9-a97f-b914a8c55c02","Type":"ContainerStarted","Data":"1ce29c3dfbd69c570c71c18496232db1cc5f5fcc08e48fc0284d0b98ca98daad"} Oct 30 19:26:28.406621 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:28.403330 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" event={"ID":"d2ceec35-f806-4418-aebd-f515b2423b4d","Type":"ContainerStarted","Data":"82617253e57025889110b7319a99da9a5288c20f2b0ab69eece691be429af7fd"} Oct 30 19:26:29.411048 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:29.410997 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" event={"ID":"97681db1-a8e8-4ad9-a97f-b914a8c55c02","Type":"ContainerStarted","Data":"3aaeb2d989d2a421c1b446bfee87458ea7d9bafa62e15fdfd746a61aa5e82443"} Oct 30 19:26:29.412735 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:29.412707 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" event={"ID":"d2ceec35-f806-4418-aebd-f515b2423b4d","Type":"ContainerStarted","Data":"5e7ecde9e7baefadb4ce1b829d0a2b381539d74ab19b67bab530c3f3d1eec9d4"} Oct 30 19:26:29.488090 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:29.486510 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" podStartSLOduration=4.486490907 podStartE2EDuration="4.486490907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:29.4863687 +0000 UTC m=+1342.560824061" watchObservedRunningTime="2026-10-30 19:26:29.486490907 +0000 UTC m=+1342.560946269" Oct 30 19:26:29.530306 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:29.530242 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" podStartSLOduration=4.530223535 podStartE2EDuration="4.530223535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:29.527132376 +0000 UTC m=+1342.601587726" watchObservedRunningTime="2026-10-30 19:26:29.530223535 +0000 UTC m=+1342.604678893" Oct 30 19:26:30.304131 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:30.304095 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:30.305173 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:30.305134 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:26:30.305317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:30.305187 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" podUID="97681db1-a8e8-4ad9-a97f-b914a8c55c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:26:30.405412 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:30.405374 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:30.406663 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:30.406633 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:26:30.406798 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:30.406688 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" podUID="d2ceec35-f806-4418-aebd-f515b2423b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:26:31.304766 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:31.304720 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:26:31.305330 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:31.304782 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" podUID="97681db1-a8e8-4ad9-a97f-b914a8c55c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:26:31.405388 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:31.405348 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:26:31.405601 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:31.405408 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" podUID="d2ceec35-f806-4418-aebd-f515b2423b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:26:32.304813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:32.304777 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:26:32.305219 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:32.304833 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" podUID="97681db1-a8e8-4ad9-a97f-b914a8c55c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:26:32.405919 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:32.405671 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:26:32.405919 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:32.405728 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" podUID="d2ceec35-f806-4418-aebd-f515b2423b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:26:33.304722 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:33.304676 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:26:33.304878 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:33.304747 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" podUID="97681db1-a8e8-4ad9-a97f-b914a8c55c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:26:33.406208 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:33.406170 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:26:33.406405 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:33.406230 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" podUID="d2ceec35-f806-4418-aebd-f515b2423b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:26:34.304451 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:34.304414 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:26:34.304790 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:34.304504 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" podUID="97681db1-a8e8-4ad9-a97f-b914a8c55c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:26:34.405359 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:34.405317 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:26:34.405750 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:34.405378 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" podUID="d2ceec35-f806-4418-aebd-f515b2423b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:26:35.304909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:35.304871 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:26:35.305123 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:35.304929 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" podUID="97681db1-a8e8-4ad9-a97f-b914a8c55c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:26:35.406196 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:35.406155 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:26:35.406686 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:35.406223 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" podUID="d2ceec35-f806-4418-aebd-f515b2423b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:26:36.487387 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:36.487349 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:36.488707 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:36.488393 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:36.489326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:36.489295 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:26:36.507173 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:36.507136 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:36.507752 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:36.507724 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:36.509292 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:36.509264 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:26:39.603137 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:26:39.603095 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:26:44.816226 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:44.816173 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:45.435429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:45.435368 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:45.524163 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:45.524113 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:45.608174 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:45.608120 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:45.817218 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:45.817111 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:26:45.818515 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:45.818489 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:26:46.439581 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:46.439534 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 19:26:46.440674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:46.440650 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:26:46.626163 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:46.626110 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:46.721086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:46.697241 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:26:47.043097 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:47.041620 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:47.774035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:47.764659 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:26:47.786954 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:47.786910 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 19:26:48.117182 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:48.117124 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:26:49.219032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:49.218098 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:26:49.219569 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:49.219412 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:26:49.249741 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:49.247132 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:49.265286 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:49.253211 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:49.449070 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:49.447318 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:26:49.449070 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:49.447395 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" podUID="709e2b00-e43d-425d-855f-8b498883c049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:26:49.475089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:49.473465 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:26:49.500831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:26:49.500795 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:27:07.124028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:07.123839 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:27:07.242032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:07.239156 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 19:27:07.548226 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:07.548120 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:07.548522 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:07.548487 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" podUID="97681db1-a8e8-4ad9-a97f-b914a8c55c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:07.622293 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:07.622243 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:07.643369 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:07.643329 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:07.643659 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:07.643628 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" podUID="d2ceec35-f806-4418-aebd-f515b2423b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:08.252029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:08.251257 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 19:27:08.321583 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:08.256790 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:27:08.694041 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:08.692811 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:27:09.604581 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:27:09.604536 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:27:28.502871 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:28.502744 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:28.695296 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:28.695240 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:28.738175 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:28.738112 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded" Oct 30 19:27:29.096039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:29.090153 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:29.663198 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:29.663137 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:29.996094 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:29.994453 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:30.544521 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:27:30.525981 5047 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.407s" Oct 30 19:27:30.544521 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:30.526154 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:27:30.544521 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:30.526197 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:27:31.012041 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.007765 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:31.012041 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.007962 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:31.467187 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.467131 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:27:31.602031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.598081 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt"] Oct 30 19:27:31.725706 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.725621 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt"] Oct 30 19:27:31.726124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.726040 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:31.837031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.836855 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:31.837248 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.837071 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:31.887117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.842151 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:31.887117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.842255 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:31.887117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.855562 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:31.887117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.855606 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:31.887117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.855650 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:31.887117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.855867 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:31.960412 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.960367 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:31.960709 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.960687 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:31.960862 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.960846 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:31.961048 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:31.960994 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:32.045989 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.045902 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:32.065584 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.065548 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:32.066018 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.065983 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:32.066184 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.066166 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b9zs\" (UniqueName: \"kubernetes.io/projected/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-kube-api-access-8b9zs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:32.066381 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.066368 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:32.066954 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.066931 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-credential-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:32.067171 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.067146 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 19:27:32.076103 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.062072 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istiod-ca-cert\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:32.076103 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.074937 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-workload-certs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:32.076103 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.045157 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-data\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:32.088213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.088173 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-workload-socket\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:32.088593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.088551 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:27:32.125059 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.115044 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-envoy\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:32.271027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.267431 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b9zs\" (UniqueName: \"kubernetes.io/projected/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-kube-api-access-8b9zs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:32.271027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.268834 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-podinfo\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:32.388521 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.354836 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b9zs\" (UniqueName: \"kubernetes.io/projected/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-kube-api-access-8b9zs\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:32.553901 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.553719 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-token\") pod \"e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:32.631229 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.631164 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:32.632404 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.632378 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:27:32.831161 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:32.738754 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:33.090428 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:33.077909 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:33.106067 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:33.096525 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:27:33.430303 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:33.430204 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:33.454059 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:33.447209 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" podUID="5c7b1347-f416-42a3-a055-0b866e894c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:33.679862 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:33.524232 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:33.681472 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:33.681387 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" podUID="0d95b5e1-e759-430f-b0fb-461451ee6e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:33.681706 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:33.656990 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:27:33.681944 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:33.681924 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:27:33.687724 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:33.574879 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:33.709141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:33.708863 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:27:34.344229 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:34.344189 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt"] Oct 30 19:27:34.345353 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:27:34.345323 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407464c4_7b3c_4c02_9bde_db3e9a0e97a7.slice/crio-d232ca23ab2b1ecd9be20acdc7afdd86523afb0ab9ff5df3f1ccf6668e77d66e WatchSource:0}: Error finding container d232ca23ab2b1ecd9be20acdc7afdd86523afb0ab9ff5df3f1ccf6668e77d66e: Status 404 returned error can't find the container with id d232ca23ab2b1ecd9be20acdc7afdd86523afb0ab9ff5df3f1ccf6668e77d66e Oct 30 19:27:34.348334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:34.348284 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:27:34.348469 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:34.348385 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:27:34.348469 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:34.348430 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:27:34.670110 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:34.669331 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" event={"ID":"407464c4-7b3c-4c02-9bde-db3e9a0e97a7","Type":"ContainerStarted","Data":"339aaa9d151381de549fbc0d1204d9c8fab4c0de6a8ce7c8692c5223eb0db804"} Oct 30 19:27:34.670110 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:34.669433 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" event={"ID":"407464c4-7b3c-4c02-9bde-db3e9a0e97a7","Type":"ContainerStarted","Data":"d232ca23ab2b1ecd9be20acdc7afdd86523afb0ab9ff5df3f1ccf6668e77d66e"} Oct 30 19:27:34.711873 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:34.711644 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" podStartSLOduration=4.711625554 podStartE2EDuration="4.711625554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:34.708750686 +0000 UTC m=+1407.783206044" watchObservedRunningTime="2026-10-30 19:27:34.711625554 +0000 UTC m=+1407.786081001" Oct 30 19:27:34.740316 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:34.740278 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:35.756041 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:35.755915 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:27:35.756041 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:35.755987 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" podUID="407464c4-7b3c-4c02-9bde-db3e9a0e97a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:27:36.747039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:36.746487 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:27:36.747039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:36.746561 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" podUID="407464c4-7b3c-4c02-9bde-db3e9a0e97a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:27:37.756029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:37.746501 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:27:37.756029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:37.746571 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" podUID="407464c4-7b3c-4c02-9bde-db3e9a0e97a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:27:38.756031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:38.755493 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:27:38.756031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:38.755564 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" podUID="407464c4-7b3c-4c02-9bde-db3e9a0e97a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:27:39.604086 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:27:39.604049 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:27:39.752029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:39.743418 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:27:39.752029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:39.743478 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" podUID="407464c4-7b3c-4c02-9bde-db3e9a0e97a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:27:40.742148 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:40.741207 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:27:40.742148 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:40.741270 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" podUID="407464c4-7b3c-4c02-9bde-db3e9a0e97a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:27:41.838324 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:41.838276 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:41.839561 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:41.839391 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:41.840779 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:41.840749 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:27:49.093705 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:49.093651 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:49.698032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:49.697557 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:27:50.072248 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:50.072136 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:50.072548 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:50.072512 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" podUID="04defd95-0e95-4ee6-9d47-d6b2488a2f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:50.077621 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:50.077579 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:50.111534 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:50.111481 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:50.114404 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:50.114372 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:27:50.706921 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:50.706873 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:27:50.708385 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:50.708357 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:27:51.095147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:51.086350 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:51.134043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:51.115317 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:51.172099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:51.133045 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:51.172099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:51.156782 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" podUID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:51.309306 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:51.306710 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:27:51.721439 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:51.721376 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:52.094037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:52.091092 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 19:27:52.181032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:52.180903 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:52.593183 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:52.593133 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:52.593496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:52.593463 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" podUID="d2ceec35-f806-4418-aebd-f515b2423b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:52.614171 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:52.614122 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:52.614534 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:52.614504 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" podUID="97681db1-a8e8-4ad9-a97f-b914a8c55c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:52.749167 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:52.749113 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:53.135029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:53.117373 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:53.697033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:53.643251 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:27:53.908721 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:53.650096 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:53.975216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:53.975110 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:27:54.118154 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:54.118104 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:54.173611 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:27:54.173573 5047 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.107s" Oct 30 19:27:54.899290 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:54.899235 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:54.927576 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:54.927540 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:27:54.952522 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:54.952122 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:54.952522 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:54.952196 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" podUID="8f5a1117-41f6-421e-a9cc-9230eea118f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:54.987407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:54.987349 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:55.143329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:55.143279 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:55.143538 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:55.143360 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" podUID="bccbf42c-7950-485f-a5b4-0475f827c419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:55.187104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:55.180239 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:55.187104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:55.180307 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" podUID="98e541fb-4108-4b0a-8c6b-7e6587354f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:55.197321 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:55.197161 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:55.197321 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:55.197243 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" podUID="586ca704-ecaf-4d51-88b4-6ea6770bb10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:55.197780 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:55.197665 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:55.197780 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:55.197723 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" podUID="708f310c-157e-4c22-8de1-1bdecacc2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:55.366028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:55.365719 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:27:55.659232 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:55.659193 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:27:56.699815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.699777 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd"] Oct 30 19:27:56.704913 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.704883 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:56.733182 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.733136 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6e0c55\"/\"openshift-service-ca.crt\"" Oct 30 19:27:56.733182 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.733158 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6e0c55\"/\"default-dockercfg-bs5pk\"" Oct 30 19:27:56.733495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.733194 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-6e0c55\"/\"kube-root-ca.crt\"" Oct 30 19:27:56.733495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.733225 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6e0c55\"/\"distinct-w6-6e0c55-kserve-self-signed-certs\"" Oct 30 19:27:56.739486 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.739445 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd"] Oct 30 19:27:56.808212 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.808170 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-dshm\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:56.808430 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.808349 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-model-cache\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:56.808430 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.808395 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-home\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:56.808571 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.808547 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-tls-certs\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:56.808863 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.808837 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-tmp-dir\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:56.808992 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.808969 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfsfj\" (UniqueName: \"kubernetes.io/projected/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-kube-api-access-zfsfj\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:56.917355 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.911121 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-tls-certs\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:56.934294 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.932560 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-tmp-dir\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:56.934294 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.932672 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfsfj\" (UniqueName: \"kubernetes.io/projected/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-kube-api-access-zfsfj\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:56.934294 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.932822 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-dshm\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:56.934294 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.932927 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-model-cache\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:56.934294 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.932963 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-home\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:56.934294 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.933606 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-home\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:56.934294 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.933886 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-tmp-dir\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:56.956085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.952778 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-model-cache\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:56.982028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.956636 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-dshm\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:56.982327 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:56.978779 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-tls-certs\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:57.019952 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:57.019916 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfsfj\" (UniqueName: \"kubernetes.io/projected/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-kube-api-access-zfsfj\") pod \"distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:57.030655 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:57.030622 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:27:57.421978 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:57.421942 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd"] Oct 30 19:27:57.422644 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:27:57.422618 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c703bcf_61b7_443b_8b9b_d09d6f11cebc.slice/crio-fbfd9ead1674c68363fadd35745b7cf3e6dee27c3fe64fce86324120d8584151 WatchSource:0}: Error finding container fbfd9ead1674c68363fadd35745b7cf3e6dee27c3fe64fce86324120d8584151: Status 404 returned error can't find the container with id fbfd9ead1674c68363fadd35745b7cf3e6dee27c3fe64fce86324120d8584151 Oct 30 19:27:57.811546 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:57.811261 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" event={"ID":"1c703bcf-61b7-443b-8b9b-d09d6f11cebc","Type":"ContainerStarted","Data":"82f5bb9a8880d816fd306137e34f425dddcfa24ed4057124ba77762ed566c504"} Oct 30 19:27:57.811546 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:57.811320 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" event={"ID":"1c703bcf-61b7-443b-8b9b-d09d6f11cebc","Type":"ContainerStarted","Data":"fbfd9ead1674c68363fadd35745b7cf3e6dee27c3fe64fce86324120d8584151"} Oct 30 19:27:57.855636 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:57.855581 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podStartSLOduration=1.85556548 podStartE2EDuration="1.85556548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:57.851656908 +0000 UTC m=+1430.926112266" watchObservedRunningTime="2026-10-30 19:27:57.85556548 +0000 UTC m=+1430.930020838" Oct 30 19:27:58.034030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:27:58.032173 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:28:07.032148 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:07.032101 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:28:07.054638 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:07.054595 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:28:07.908039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:07.907983 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:28:09.638494 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:28:09.638452 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:28:11.766188 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:11.766123 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:11.967190 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:11.967126 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:12.168276 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:12.168223 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:12.778034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:12.777108 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:12.782030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:12.779439 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:28:12.926027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:12.925222 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:13.076866 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:13.076758 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 19:28:13.202477 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:13.197192 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:13.324030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:13.321759 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:28:13.324030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:13.321826 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:28:13.824032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:13.823152 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:13.935039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:13.934623 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:13.939031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:13.935628 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:28:14.130063 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:14.129995 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 19:28:14.231056 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:14.219567 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:14.878431 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:14.877330 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:14.935032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:14.932764 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:28:14.935347 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:14.935311 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 19:28:15.284361 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:15.284257 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:15.912164 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:15.912108 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:15.936029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:15.934478 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:15.936377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:15.936339 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 19:28:16.326556 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:16.323361 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:16.369030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:16.351207 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:28:16.462032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:16.451358 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:28:16.920842 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:16.920797 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:28:17.126730 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:16.956158 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:17.210181 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:17.210093 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:28:17.210861 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:17.210841 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:28:17.213086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:17.213064 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:28:31.227551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.227507 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg"] Oct 30 19:28:31.234903 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.234875 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.261791 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.261752 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg"] Oct 30 19:28:31.271143 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.271114 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j"] Oct 30 19:28:31.275880 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.275854 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.305727 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.305692 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j"] Oct 30 19:28:31.358117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358078 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.358335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358133 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c080477b-5afb-4972-8feb-1bd6ec66092f-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.358335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358185 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.358335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358285 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.358335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358330 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.358594 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358348 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.358594 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358428 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.358594 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358453 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.358594 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358495 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.358594 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358543 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.358594 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358567 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.358909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358615 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.358909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358692 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.358909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358769 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.358909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358811 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.358909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358832 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84qtt\" (UniqueName: \"kubernetes.io/projected/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-kube-api-access-84qtt\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.358909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358885 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppz85\" (UniqueName: \"kubernetes.io/projected/c080477b-5afb-4972-8feb-1bd6ec66092f-kube-api-access-ppz85\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.358909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.358900 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.460555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.460516 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.460744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.460567 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.460744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.460606 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.460744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.460629 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.460744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.460648 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.460744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.460671 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.460991 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.460766 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.460991 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.460807 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.460991 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.460861 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.460991 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.460889 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84qtt\" (UniqueName: \"kubernetes.io/projected/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-kube-api-access-84qtt\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.460991 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.460935 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppz85\" (UniqueName: \"kubernetes.io/projected/c080477b-5afb-4972-8feb-1bd6ec66092f-kube-api-access-ppz85\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.460991 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.460962 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.461305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.461077 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.461305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.461090 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.461305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.461119 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c080477b-5afb-4972-8feb-1bd6ec66092f-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.461305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.461090 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.461305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.461116 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.461305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.461201 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.461305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.461256 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.461657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.461315 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.461657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.461330 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.461657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.461340 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.461657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.461401 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.461657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.461587 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.461964 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.461736 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.461964 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.461944 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.462155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.462133 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.462688 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.462659 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c080477b-5afb-4972-8feb-1bd6ec66092f-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.463410 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.463387 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.463628 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.463602 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.463957 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.463935 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.464397 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.464378 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.481277 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.481197 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.481425 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.481310 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84qtt\" (UniqueName: \"kubernetes.io/projected/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-kube-api-access-84qtt\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.481493 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.481458 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppz85\" (UniqueName: \"kubernetes.io/projected/c080477b-5afb-4972-8feb-1bd6ec66092f-kube-api-access-ppz85\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.481493 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.481468 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.550019 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.549970 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:31.599918 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.599881 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:31.870411 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.870377 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg"] Oct 30 19:28:31.870530 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:28:31.870386 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc080477b_5afb_4972_8feb_1bd6ec66092f.slice/crio-0ce41d883babf8f481e8f2837f685e11cc5709b93fdf75be1384130c6bf3b4ba WatchSource:0}: Error finding container 0ce41d883babf8f481e8f2837f685e11cc5709b93fdf75be1384130c6bf3b4ba: Status 404 returned error can't find the container with id 0ce41d883babf8f481e8f2837f685e11cc5709b93fdf75be1384130c6bf3b4ba Oct 30 19:28:31.872875 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.872846 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:28:31.872982 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.872913 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:28:31.872982 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.872942 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:28:31.961981 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:28:31.961954 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c276d2_2b71_4701_a4f2_bf3dbc9f9a75.slice/crio-7b42125b7b65cc213bf77ff15c39de10490e18f6ec608251b2538c04ed03a605 WatchSource:0}: Error finding container 7b42125b7b65cc213bf77ff15c39de10490e18f6ec608251b2538c04ed03a605: Status 404 returned error can't find the container with id 7b42125b7b65cc213bf77ff15c39de10490e18f6ec608251b2538c04ed03a605 Oct 30 19:28:31.962178 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.962155 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j"] Oct 30 19:28:31.964666 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.964637 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:28:31.964756 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.964698 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:28:31.964756 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:31.964728 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:28:32.296045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:32.295980 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" event={"ID":"c080477b-5afb-4972-8feb-1bd6ec66092f","Type":"ContainerStarted","Data":"42d44f42b473a9756449131fd12d14aafddc51b3e4764864c0084e4c208d1229"} Oct 30 19:28:32.296496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:32.296057 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" event={"ID":"c080477b-5afb-4972-8feb-1bd6ec66092f","Type":"ContainerStarted","Data":"0ce41d883babf8f481e8f2837f685e11cc5709b93fdf75be1384130c6bf3b4ba"} Oct 30 19:28:32.298048 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:32.298017 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" event={"ID":"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75","Type":"ContainerStarted","Data":"343cbb955797d7758c8589372d766e6cdc279a36fb8dee3c78c5b9efb04b8c66"} Oct 30 19:28:32.298180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:32.298074 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" event={"ID":"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75","Type":"ContainerStarted","Data":"7b42125b7b65cc213bf77ff15c39de10490e18f6ec608251b2538c04ed03a605"} Oct 30 19:28:32.337848 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:32.337794 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" podStartSLOduration=1.337780668 podStartE2EDuration="1.337780668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:32.336283007 +0000 UTC m=+1465.410738367" watchObservedRunningTime="2026-10-30 19:28:32.337780668 +0000 UTC m=+1465.412236088" Oct 30 19:28:32.377599 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:32.377533 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" podStartSLOduration=1.377511892 podStartE2EDuration="1.377511892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:32.376083563 +0000 UTC m=+1465.450538923" watchObservedRunningTime="2026-10-30 19:28:32.377511892 +0000 UTC m=+1465.451967253" Oct 30 19:28:32.550938 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:32.550835 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:32.552873 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:32.552844 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:28:32.553061 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:32.552937 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:28:32.600494 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:32.600462 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:33.571031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:33.555686 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:28:33.571031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:33.555752 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:28:33.601911 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:33.601868 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:28:33.602146 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:33.601929 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" podUID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:28:34.553029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:34.550919 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:28:34.553029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:34.550984 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:28:34.601221 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:34.601184 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:28:34.601634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:34.601240 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" podUID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:28:35.556180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:35.556136 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:28:35.557113 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:35.557075 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:28:35.601259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:35.601217 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:28:35.601651 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:35.601287 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" podUID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:28:36.561792 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:36.561752 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:28:36.562087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:36.562059 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:28:36.636509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:36.636473 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:28:36.662394 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:36.637095 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" podUID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:28:37.230031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:37.229901 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:37.230247 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:37.230099 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:28:37.554090 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:37.553478 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:28:37.554090 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:37.553546 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:28:37.613031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:37.610364 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:28:37.613031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:37.610434 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" podUID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:28:37.838625 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:37.837124 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:28:37.950526 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:37.941123 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 19:28:38.042030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:38.041474 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:38.234235 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:38.234105 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:38.235335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:38.235308 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:28:38.246169 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:38.246124 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:38.248184 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:38.248133 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:28:38.577090 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:38.563025 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:28:38.577090 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:38.563098 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:28:38.624298 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:38.624157 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:28:38.624298 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:38.624229 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" podUID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:28:38.754057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:38.753924 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded" Oct 30 19:28:38.833093 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:38.811879 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f"] Oct 30 19:28:38.857910 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:38.839089 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:28:38.866185 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:38.866141 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:28:38.866775 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:38.866740 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f"] Oct 30 19:28:38.868159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:38.868093 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:38.922032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:38.921924 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-6e0c55\"/\"distinct-2-w6-6e0c55-kserve-self-signed-certs\"" Oct 30 19:28:39.165605 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:39.165486 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:39.747055 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:39.737610 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:39.847038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:39.834688 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:28:39.868552 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:39.861571 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:28:39.873043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:39.868171 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:28:39.916305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:39.915141 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" podUID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:28:40.028998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:39.861652 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:28:40.043213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.043167 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:40.043519 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.043483 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" podUID="98e541fb-4108-4b0a-8c6b-7e6587354f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.061056 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.044312 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.202077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.191137 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.202077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.191302 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:40.202077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.191352 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" podUID="bccbf42c-7950-485f-a5b4-0475f827c419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.273402 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.227150 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:40.273402 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.227239 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" podUID="8f5a1117-41f6-421e-a9cc-9230eea118f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.273402 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.228544 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 19:28:40.273402 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.272754 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-home\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:40.273402 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.272804 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-tmp-dir\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:40.273402 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.273025 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-model-cache\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:40.273402 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.273176 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-tls-certs\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:40.273402 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.273290 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mgbn\" (UniqueName: \"kubernetes.io/projected/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-kube-api-access-8mgbn\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:40.273402 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.273364 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-dshm\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:40.417029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.366157 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:40.417029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.366226 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" podUID="586ca704-ecaf-4d51-88b4-6ea6770bb10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.462189 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.462132 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg" podUID="d6cf3827-a5fe-48e4-900a-1f1633ff69ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.524032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.498777 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-model-cache\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:40.596611 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.596570 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-model-cache\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:40.597991 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.597961 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-tls-certs\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:40.602031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.598274 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mgbn\" (UniqueName: \"kubernetes.io/projected/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-kube-api-access-8mgbn\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:40.602305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.602281 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-dshm\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:40.606036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.602524 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-home\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:40.606207 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.606186 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-tmp-dir\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:40.620030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.611631 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-tmp-dir\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:40.701181 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.632892 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:28:40.701181 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.650551 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:28:40.701181 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.650620 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" podUID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:28:40.701181 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.693120 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:28:40.800884 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.745375 5047 patch_prober.go:29] interesting pod/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:40.800884 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.745438 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" podUID="708f310c-157e-4c22-8de1-1bdecacc2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.800884 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.789422 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:40.898025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.897969 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-tls-certs\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:40.954494 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:40.954411 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-home\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:41.026031 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:28:40.991646 5047 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.157s" Oct 30 19:28:41.070147 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:28:41.070113 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:28:41.229353 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:41.089033 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-dshm\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:41.229353 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:41.065550 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:28:41.229353 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:41.228955 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:41.253054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:41.230897 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:41.253054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:41.230965 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" podUID="9399ee4d-b327-4739-85b1-d18eecea8116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:41.394087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:41.394031 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 19:28:41.609030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:41.604610 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mgbn\" (UniqueName: \"kubernetes.io/projected/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-kube-api-access-8mgbn\") pod \"distinct-2-w6-6e0c55-kserve-7f8d856b-w979f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:41.646031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:41.643566 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:41.673282 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:41.673242 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:28:41.673619 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:41.673567 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:28:41.730434 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:41.730386 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:28:41.730950 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:41.730912 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" podUID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:28:41.795608 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:41.795556 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:42.223377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:42.223316 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:28:42.229350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:42.229306 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:42.415064 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:42.394451 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:42.578809 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:42.575742 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:28:42.578809 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:42.575805 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:28:42.676488 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:42.676377 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:28:42.676488 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:42.676460 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" podUID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:28:42.820028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:42.801206 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:28:42.952715 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:42.952247 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:28:42.952715 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:42.952316 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" podUID="407464c4-7b3c-4c02-9bde-db3e9a0e97a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:43.241601 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:43.230358 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:28:43.250250 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:43.230315 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:43.504110 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:43.494238 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 19:28:43.577383 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:43.577334 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:43.642427 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:43.642389 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f"] Oct 30 19:28:43.730060 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:43.726170 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:28:43.730060 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:43.726233 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:28:43.783089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:43.767889 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:28:43.783089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:43.767952 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" podUID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:28:43.821268 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:43.819118 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:43.967413 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:43.967381 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:28:44.129204 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:44.129165 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" event={"ID":"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f","Type":"ContainerStarted","Data":"9c117754aff20075158a73384a033cce5c5521468c371503a144653eb08f076f"} Oct 30 19:28:44.203379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:44.203353 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:28:44.268394 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:44.253707 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:28:44.268394 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:44.253778 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:28:44.550997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:44.550965 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:28:44.551235 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:44.551213 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:28:44.600488 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:44.600437 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:28:44.600692 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:44.600506 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" podUID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:28:45.138832 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:45.138793 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" event={"ID":"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f","Type":"ContainerStarted","Data":"32feb7913287cdf8e8820e7989f5e76ba00093c1dfab6c012277959a51c9ddde"} Oct 30 19:28:45.554938 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:45.554862 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:45.555344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:45.555255 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:45.555988 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:45.555971 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:28:45.599360 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:45.599301 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podStartSLOduration=7.599283423 podStartE2EDuration="7.599283423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:45.186224223 +0000 UTC m=+1478.260679584" watchObservedRunningTime="2026-10-30 19:28:45.599283423 +0000 UTC m=+1478.673738785" Oct 30 19:28:45.605584 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:45.605559 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:45.605861 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:45.605835 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:45.606679 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:45.606659 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:28:45.668957 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:45.668918 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:51.653697 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:51.653612 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx"] Oct 30 19:28:51.658164 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:51.658142 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" Oct 30 19:28:51.668750 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:51.668720 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:51.672957 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:51.672933 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d4245aa3-dockercfg-nhp7w\"" Oct 30 19:28:51.683987 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:51.683954 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:51.684731 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:51.684706 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d4245aa3\"" Oct 30 19:28:51.690857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:51.690827 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx"] Oct 30 19:28:51.854836 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:51.834087 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb"] Oct 30 19:28:51.859633 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:51.858365 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k746v\" (UniqueName: \"kubernetes.io/projected/d6a1ea21-bd4e-4f7b-8d35-737204ee704d-kube-api-access-k746v\") pod \"payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx\" (UID: \"d6a1ea21-bd4e-4f7b-8d35-737204ee704d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" Oct 30 19:28:51.859633 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:51.858725 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a1ea21-bd4e-4f7b-8d35-737204ee704d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx\" (UID: \"d6a1ea21-bd4e-4f7b-8d35-737204ee704d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" Oct 30 19:28:51.910079 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:51.909343 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" Oct 30 19:28:52.085954 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:52.002210 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k746v\" (UniqueName: \"kubernetes.io/projected/d6a1ea21-bd4e-4f7b-8d35-737204ee704d-kube-api-access-k746v\") pod \"payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx\" (UID: \"d6a1ea21-bd4e-4f7b-8d35-737204ee704d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" Oct 30 19:28:52.085954 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:52.002333 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9584986-cf2e-47b2-af7a-16edc82d769a-plugins-config-volume\") pod \"payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb\" (UID: \"c9584986-cf2e-47b2-af7a-16edc82d769a\") " pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" Oct 30 19:28:52.085954 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:52.002528 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a1ea21-bd4e-4f7b-8d35-737204ee704d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx\" (UID: \"d6a1ea21-bd4e-4f7b-8d35-737204ee704d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" Oct 30 19:28:52.085954 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:52.002562 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnjgc\" (UniqueName: \"kubernetes.io/projected/c9584986-cf2e-47b2-af7a-16edc82d769a-kube-api-access-qnjgc\") pod \"payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb\" (UID: \"c9584986-cf2e-47b2-af7a-16edc82d769a\") " pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" Oct 30 19:28:52.085954 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:52.018472 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb"] Oct 30 19:28:52.085954 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:52.056848 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k746v\" (UniqueName: \"kubernetes.io/projected/d6a1ea21-bd4e-4f7b-8d35-737204ee704d-kube-api-access-k746v\") pod \"payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx\" (UID: \"d6a1ea21-bd4e-4f7b-8d35-737204ee704d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" Oct 30 19:28:52.085954 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:52.084590 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a1ea21-bd4e-4f7b-8d35-737204ee704d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx\" (UID: \"d6a1ea21-bd4e-4f7b-8d35-737204ee704d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" Oct 30 19:28:52.117702 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:52.117662 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9584986-cf2e-47b2-af7a-16edc82d769a-plugins-config-volume\") pod \"payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb\" (UID: \"c9584986-cf2e-47b2-af7a-16edc82d769a\") " pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" Oct 30 19:28:52.118057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:52.118037 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnjgc\" (UniqueName: \"kubernetes.io/projected/c9584986-cf2e-47b2-af7a-16edc82d769a-kube-api-access-qnjgc\") pod \"payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb\" (UID: \"c9584986-cf2e-47b2-af7a-16edc82d769a\") " pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" Oct 30 19:28:52.119324 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:52.118702 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9584986-cf2e-47b2-af7a-16edc82d769a-plugins-config-volume\") pod \"payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb\" (UID: \"c9584986-cf2e-47b2-af7a-16edc82d769a\") " pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" Oct 30 19:28:52.137386 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:52.137348 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnjgc\" (UniqueName: \"kubernetes.io/projected/c9584986-cf2e-47b2-af7a-16edc82d769a-kube-api-access-qnjgc\") pod \"payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb\" (UID: \"c9584986-cf2e-47b2-af7a-16edc82d769a\") " pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" Oct 30 19:28:52.271278 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:52.271191 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:28:52.272428 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:52.272408 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" Oct 30 19:28:52.388961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:52.388928 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" Oct 30 19:28:52.585436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:52.585408 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx"] Oct 30 19:28:52.586600 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:28:52.586555 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a1ea21_bd4e_4f7b_8d35_737204ee704d.slice/crio-c7aa074e4c69ad0b7bba11427787e30e237628b1eb464ccf473a146980867cbf WatchSource:0}: Error finding container c7aa074e4c69ad0b7bba11427787e30e237628b1eb464ccf473a146980867cbf: Status 404 returned error can't find the container with id c7aa074e4c69ad0b7bba11427787e30e237628b1eb464ccf473a146980867cbf Oct 30 19:28:52.727186 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:52.727160 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb"] Oct 30 19:28:52.727812 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:28:52.727787 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9584986_cf2e_47b2_af7a_16edc82d769a.slice/crio-d8c19c4803afbda30ad5dbd4fbfc8060b14e957c5a6fef5cf5f810eb4fe670e5 WatchSource:0}: Error finding container d8c19c4803afbda30ad5dbd4fbfc8060b14e957c5a6fef5cf5f810eb4fe670e5: Status 404 returned error can't find the container with id d8c19c4803afbda30ad5dbd4fbfc8060b14e957c5a6fef5cf5f810eb4fe670e5 Oct 30 19:28:53.268333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:53.268297 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" event={"ID":"c9584986-cf2e-47b2-af7a-16edc82d769a","Type":"ContainerStarted","Data":"61144f283904c689716a5b406bdc75f2b94f6fcbcf5e4bb5f4a49464af3b1b38"} Oct 30 19:28:53.268555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:53.268370 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" event={"ID":"c9584986-cf2e-47b2-af7a-16edc82d769a","Type":"ContainerStarted","Data":"d8c19c4803afbda30ad5dbd4fbfc8060b14e957c5a6fef5cf5f810eb4fe670e5"} Oct 30 19:28:53.268555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:53.268507 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" Oct 30 19:28:53.274788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:53.274754 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" event={"ID":"d6a1ea21-bd4e-4f7b-8d35-737204ee704d","Type":"ContainerStarted","Data":"a05107fd86f7acbcf39bc42a7e0fbd275ff58ca315c84d1fef764441b5194946"} Oct 30 19:28:53.274953 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:53.274813 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" event={"ID":"d6a1ea21-bd4e-4f7b-8d35-737204ee704d","Type":"ContainerStarted","Data":"c7aa074e4c69ad0b7bba11427787e30e237628b1eb464ccf473a146980867cbf"} Oct 30 19:28:53.274953 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:53.274900 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" Oct 30 19:28:53.308976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:53.308915 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" podStartSLOduration=2.308896135 podStartE2EDuration="2.308896135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:53.306035364 +0000 UTC m=+1486.380490722" watchObservedRunningTime="2026-10-30 19:28:53.308896135 +0000 UTC m=+1486.383351494" Oct 30 19:28:53.363959 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:28:53.363905 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" podStartSLOduration=2.363887848 podStartE2EDuration="2.363887848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:53.359241655 +0000 UTC m=+1486.433697018" watchObservedRunningTime="2026-10-30 19:28:53.363887848 +0000 UTC m=+1486.438343240" Oct 30 19:29:01.613111 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.613068 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c"] Oct 30 19:29:01.672050 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.650345 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.686043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.675606 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.701291 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.701253 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.701678 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.701650 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.701933 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.701911 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.714031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.706159 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.714489 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.714452 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.722238 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.722185 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.722556 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.722533 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.743337 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.743297 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9sxf\" (UniqueName: \"kubernetes.io/projected/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-kube-api-access-b9sxf\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.744617 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.744585 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c"] Oct 30 19:29:01.809694 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.809645 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:01.876104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.864999 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.876104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.865100 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.876104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.865214 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.876104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.865272 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.876104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.865308 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.876104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.865395 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.876104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.865498 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.876104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.865539 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.876104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.865611 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9sxf\" (UniqueName: \"kubernetes.io/projected/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-kube-api-access-b9sxf\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.877977 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.877942 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-data\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.878466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.878439 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-credential-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.879267 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.879241 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-workload-socket\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.902167 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.902117 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istiod-ca-cert\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:01.928785 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:01.919215 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-workload-certs\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:02.042032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:02.008268 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-envoy\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:02.049031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:02.048894 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9sxf\" (UniqueName: \"kubernetes.io/projected/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-kube-api-access-b9sxf\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:02.074031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:02.050172 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.202085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:02.155668 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-podinfo\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:02.251053 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:02.247941 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.283057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:02.267103 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.284831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:02.284558 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-token\") pod \"e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:02.400340 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:02.400295 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.414830 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:02.414578 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:02.687806 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:02.687465 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.859035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:02.837673 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:02.859035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:02.838199 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:29:03.125884 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:03.125761 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:03.309042 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:03.253362 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 19:29:03.358273 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:03.357351 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:03.446947 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:03.446819 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 19:29:03.847674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:03.839920 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 30 19:29:04.017825 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:04.005167 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:29:04.186969 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:04.166441 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:29:04.384149 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:04.359993 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:04.506507 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:04.506387 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:04.565586 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:04.565533 5047 patch_prober.go:29] interesting pod/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:29:04.573869 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:04.573827 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" podUID="709e2b00-e43d-425d-855f-8b498883c049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:04.595314 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:04.595262 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.039039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:05.013614 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.175892 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:05.175830 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:29:05.540209 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:05.540165 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.782031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:05.742685 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.782031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:05.743092 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 19:29:05.782031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:05.744085 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 30 19:29:05.827081 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:05.826960 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g" podUID="1b105e8c-2f71-484d-b6fa-454fdc1dc3e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": context deadline exceeded" Oct 30 19:29:05.827896 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:29:05.827869 5047 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.767s" Oct 30 19:29:05.848713 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:05.848660 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:29:05.870092 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:05.861266 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" podUID="04defd95-0e95-4ee6-9d47-d6b2488a2f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:05.931068 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:05.930965 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:29:05.931271 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:05.931096 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:29:05.931271 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:05.931146 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:29:06.226517 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:06.226459 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:06.525261 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:06.525180 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:29:06.525562 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:06.525543 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:29:06.525749 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:06.525727 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:29:06.525937 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:06.525914 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:29:06.526124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:06.526101 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" Oct 30 19:29:06.526316 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:06.526304 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" Oct 30 19:29:06.526504 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:06.526476 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:29:06.530125 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:06.530092 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c"] Oct 30 19:29:06.530663 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:06.530597 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:29:06.552653 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:06.552620 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:29:06.607619 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:06.606063 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:29:06.936884 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:06.935079 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:29:07.229180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.229088 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" event={"ID":"d0e81adf-2dbf-41ff-a413-13ddc4310c6c","Type":"ContainerStarted","Data":"66548573291e74450d87f4fded492f635601cf85b59058f1672766cc7948a7fc"} Oct 30 19:29:07.240063 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.233959 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" event={"ID":"d0e81adf-2dbf-41ff-a413-13ddc4310c6c","Type":"ContainerStarted","Data":"e5d58ea678fac790ff19b923695b4bb92294608803b2ba52e55864bcb8597763"} Oct 30 19:29:07.347931 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.347905 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:29:07.353442 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.353413 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:29:07.368815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.368756 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" podStartSLOduration=6.36873621 podStartE2EDuration="6.36873621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:07.367405964 +0000 UTC m=+1500.441861324" watchObservedRunningTime="2026-10-30 19:29:07.36873621 +0000 UTC m=+1500.443191570" Oct 30 19:29:07.438415 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.438354 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:07.663760 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.663678 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54"] Oct 30 19:29:07.674479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.674446 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:07.700124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.690618 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d4245aa3-serving-cert\"" Oct 30 19:29:07.706042 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.705938 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54"] Oct 30 19:29:07.758247 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.758207 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/401ee1cc-c3eb-4564-8429-df95be765de8-metrics-certs\") pod \"maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54\" (UID: \"401ee1cc-c3eb-4564-8429-df95be765de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:07.758593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.758570 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctn9b\" (UniqueName: \"kubernetes.io/projected/401ee1cc-c3eb-4564-8429-df95be765de8-kube-api-access-ctn9b\") pod \"maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54\" (UID: \"401ee1cc-c3eb-4564-8429-df95be765de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:07.758768 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.758754 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401ee1cc-c3eb-4564-8429-df95be765de8-maas-api-tls\") pod \"maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54\" (UID: \"401ee1cc-c3eb-4564-8429-df95be765de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:07.758902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.758888 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401ee1cc-c3eb-4564-8429-df95be765de8-openshift-service-ca\") pod \"maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54\" (UID: \"401ee1cc-c3eb-4564-8429-df95be765de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:07.862844 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.861071 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/401ee1cc-c3eb-4564-8429-df95be765de8-metrics-certs\") pod \"maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54\" (UID: \"401ee1cc-c3eb-4564-8429-df95be765de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:07.862844 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.861140 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctn9b\" (UniqueName: \"kubernetes.io/projected/401ee1cc-c3eb-4564-8429-df95be765de8-kube-api-access-ctn9b\") pod \"maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54\" (UID: \"401ee1cc-c3eb-4564-8429-df95be765de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:07.862844 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.861191 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401ee1cc-c3eb-4564-8429-df95be765de8-maas-api-tls\") pod \"maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54\" (UID: \"401ee1cc-c3eb-4564-8429-df95be765de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:07.862844 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.861222 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401ee1cc-c3eb-4564-8429-df95be765de8-openshift-service-ca\") pod \"maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54\" (UID: \"401ee1cc-c3eb-4564-8429-df95be765de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:07.862844 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.862412 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401ee1cc-c3eb-4564-8429-df95be765de8-openshift-service-ca\") pod \"maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54\" (UID: \"401ee1cc-c3eb-4564-8429-df95be765de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:07.867342 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.867314 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/401ee1cc-c3eb-4564-8429-df95be765de8-metrics-certs\") pod \"maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54\" (UID: \"401ee1cc-c3eb-4564-8429-df95be765de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:07.867517 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.867323 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401ee1cc-c3eb-4564-8429-df95be765de8-maas-api-tls\") pod \"maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54\" (UID: \"401ee1cc-c3eb-4564-8429-df95be765de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:07.885830 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:07.885792 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctn9b\" (UniqueName: \"kubernetes.io/projected/401ee1cc-c3eb-4564-8429-df95be765de8-kube-api-access-ctn9b\") pod \"maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54\" (UID: \"401ee1cc-c3eb-4564-8429-df95be765de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:08.009092 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:08.005684 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:08.417538 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:08.417500 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:29:08.417966 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:08.417565 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" podUID="d0e81adf-2dbf-41ff-a413-13ddc4310c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 19:29:08.564323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:08.562951 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54"] Oct 30 19:29:08.565864 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:29:08.565831 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401ee1cc_c3eb_4564_8429_df95be765de8.slice/crio-c4e0baa769a5b186ba7ea7074c0fe5285c86df894ae757e3f2f2c48109bde26d WatchSource:0}: Error finding container c4e0baa769a5b186ba7ea7074c0fe5285c86df894ae757e3f2f2c48109bde26d: Status 404 returned error can't find the container with id c4e0baa769a5b186ba7ea7074c0fe5285c86df894ae757e3f2f2c48109bde26d Oct 30 19:29:08.567447 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:08.567427 5047 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:29:09.164045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.163967 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2"] Oct 30 19:29:09.168746 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.168717 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" Oct 30 19:29:09.186050 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.182903 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c5e08e43\"" Oct 30 19:29:09.186050 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.183329 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c5e08e43-dockercfg-sr8lg\"" Oct 30 19:29:09.188106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.188081 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2"] Oct 30 19:29:09.440676 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.424774 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d066642d-6752-4173-859b-2ed57ecbba40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2\" (UID: \"d066642d-6752-4173-859b-2ed57ecbba40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" Oct 30 19:29:09.440676 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.424914 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwvls\" (UniqueName: \"kubernetes.io/projected/d066642d-6752-4173-859b-2ed57ecbba40-kube-api-access-mwvls\") pod \"payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2\" (UID: \"d066642d-6752-4173-859b-2ed57ecbba40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" Oct 30 19:29:09.440676 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.436207 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:29:09.440676 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.436277 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" podUID="d0e81adf-2dbf-41ff-a413-13ddc4310c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 19:29:09.581602 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.581551 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d066642d-6752-4173-859b-2ed57ecbba40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2\" (UID: \"d066642d-6752-4173-859b-2ed57ecbba40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" Oct 30 19:29:09.581958 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.581931 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwvls\" (UniqueName: \"kubernetes.io/projected/d066642d-6752-4173-859b-2ed57ecbba40-kube-api-access-mwvls\") pod \"payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2\" (UID: \"d066642d-6752-4173-859b-2ed57ecbba40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" Oct 30 19:29:09.689919 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.687739 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d066642d-6752-4173-859b-2ed57ecbba40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2\" (UID: \"d066642d-6752-4173-859b-2ed57ecbba40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" Oct 30 19:29:09.723902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.723810 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwvls\" (UniqueName: \"kubernetes.io/projected/d066642d-6752-4173-859b-2ed57ecbba40-kube-api-access-mwvls\") pod \"payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2\" (UID: \"d066642d-6752-4173-859b-2ed57ecbba40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" Oct 30 19:29:09.789049 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.781648 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" event={"ID":"401ee1cc-c3eb-4564-8429-df95be765de8","Type":"ContainerStarted","Data":"2c570a086b3dc3720811ca4a22ceb5632d6832b3e7236ab1198f6e0328c9a3af"} Oct 30 19:29:09.789049 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.781718 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" event={"ID":"401ee1cc-c3eb-4564-8429-df95be765de8","Type":"ContainerStarted","Data":"c4e0baa769a5b186ba7ea7074c0fe5285c86df894ae757e3f2f2c48109bde26d"} Oct 30 19:29:09.789049 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.781776 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:09.792274 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.792236 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" Oct 30 19:29:09.826911 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:09.826826 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" podStartSLOduration=2.423956863 podStartE2EDuration="2.826803664s" totalImagesPullingTime="402.846801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:09.82432765 +0000 UTC m=+1502.898783006" watchObservedRunningTime="2026-10-30 19:29:09.826803664 +0000 UTC m=+1502.901259025" Oct 30 19:29:09.870670 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:29:09.870635 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:29:10.416645 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:10.416600 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:29:10.416834 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:10.416672 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" podUID="d0e81adf-2dbf-41ff-a413-13ddc4310c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 19:29:10.621794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:10.621756 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2"] Oct 30 19:29:10.774327 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:10.774293 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" event={"ID":"d066642d-6752-4173-859b-2ed57ecbba40","Type":"ContainerStarted","Data":"0ff68acf4427b7f2a35f994abdb0bc809833f3f479469bac4d89713e2647fa41"} Oct 30 19:29:11.417541 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:11.417495 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:29:11.417856 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:11.417816 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" podUID="d0e81adf-2dbf-41ff-a413-13ddc4310c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 19:29:11.783480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:11.783384 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" event={"ID":"d066642d-6752-4173-859b-2ed57ecbba40","Type":"ContainerStarted","Data":"a7b6a77688cec6cff6be8b55b98e2e9ce175f590846752c7914de7aa9929dbd7"} Oct 30 19:29:11.783881 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:11.783485 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" Oct 30 19:29:11.821348 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:11.821292 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" podStartSLOduration=2.821270865 podStartE2EDuration="2.821270865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:11.82059331 +0000 UTC m=+1504.895048660" watchObservedRunningTime="2026-10-30 19:29:11.821270865 +0000 UTC m=+1504.895726232" Oct 30 19:29:12.424029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:12.420506 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:29:12.424029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:12.420572 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" podUID="d0e81adf-2dbf-41ff-a413-13ddc4310c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 19:29:13.416374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:13.416340 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:29:13.416883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:13.416400 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" podUID="d0e81adf-2dbf-41ff-a413-13ddc4310c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 19:29:14.415962 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:14.415921 5047 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:29:14.416192 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:14.415986 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" podUID="d0e81adf-2dbf-41ff-a413-13ddc4310c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 19:29:15.420635 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:15.420602 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:15.421135 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:15.421103 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:15.421902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:15.421877 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:29:16.790479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:16.790449 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:19.567034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:19.566108 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:19.946033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:19.945854 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:29:20.015159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:20.015107 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:29:20.075186 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:20.075147 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:29:20.075485 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:20.075453 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" podUID="04defd95-0e95-4ee6-9d47-d6b2488a2f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:20.257227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:20.257115 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:20.303032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:20.302359 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 30 19:29:20.568230 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:20.568126 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:20.580337 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:20.580301 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:29:20.955034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:20.954180 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:29:20.985837 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:20.957532 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:29:21.051744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:21.020202 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:29:21.052187 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:21.052150 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:29:21.295978 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:21.257666 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 19:29:21.305141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:21.266245 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:29:21.305141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:21.301403 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" podUID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:21.324033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:21.308799 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:21.572225 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:21.572110 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:21.870035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:21.864357 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:29:21.870035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:21.864426 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:29:21.955080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:21.954995 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:22.067621 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:22.067089 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:29:22.357028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:22.345779 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:22.357028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:22.354501 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:22.474137 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:22.470986 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:29:22.573023 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:22.572957 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:22.804494 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:22.804377 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:22.918403 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:22.918364 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" Oct 30 19:29:22.947541 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:22.945566 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:29:22.983453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:22.982019 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:29:22.983453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:22.982883 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:29:23.033028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:23.032868 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:29:26.640939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:26.640904 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl"] Oct 30 19:29:26.649043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:26.648974 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:29:26.672312 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:26.672281 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c5e08e43-serving-cert\"" Oct 30 19:29:26.674586 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:26.674553 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl"] Oct 30 19:29:26.774784 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:26.774739 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28551278-c831-45af-8fbe-a9c7155f1801-maas-api-tls\") pod \"maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl\" (UID: \"28551278-c831-45af-8fbe-a9c7155f1801\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:29:26.774990 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:26.774963 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28551278-c831-45af-8fbe-a9c7155f1801-metrics-certs\") pod \"maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl\" (UID: \"28551278-c831-45af-8fbe-a9c7155f1801\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:29:26.775113 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:26.775059 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc2ps\" (UniqueName: \"kubernetes.io/projected/28551278-c831-45af-8fbe-a9c7155f1801-kube-api-access-hc2ps\") pod \"maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl\" (UID: \"28551278-c831-45af-8fbe-a9c7155f1801\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:29:26.775244 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:26.775224 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28551278-c831-45af-8fbe-a9c7155f1801-openshift-service-ca\") pod \"maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl\" (UID: \"28551278-c831-45af-8fbe-a9c7155f1801\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:29:26.877237 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:26.877200 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28551278-c831-45af-8fbe-a9c7155f1801-metrics-certs\") pod \"maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl\" (UID: \"28551278-c831-45af-8fbe-a9c7155f1801\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:29:26.877449 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:26.877262 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc2ps\" (UniqueName: \"kubernetes.io/projected/28551278-c831-45af-8fbe-a9c7155f1801-kube-api-access-hc2ps\") pod \"maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl\" (UID: \"28551278-c831-45af-8fbe-a9c7155f1801\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:29:26.877449 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:26.877333 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28551278-c831-45af-8fbe-a9c7155f1801-openshift-service-ca\") pod \"maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl\" (UID: \"28551278-c831-45af-8fbe-a9c7155f1801\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:29:26.877566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:26.877510 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28551278-c831-45af-8fbe-a9c7155f1801-maas-api-tls\") pod \"maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl\" (UID: \"28551278-c831-45af-8fbe-a9c7155f1801\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:29:26.878204 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:26.878177 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28551278-c831-45af-8fbe-a9c7155f1801-openshift-service-ca\") pod \"maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl\" (UID: \"28551278-c831-45af-8fbe-a9c7155f1801\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:29:26.880456 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:26.880426 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28551278-c831-45af-8fbe-a9c7155f1801-maas-api-tls\") pod \"maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl\" (UID: \"28551278-c831-45af-8fbe-a9c7155f1801\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:29:26.880832 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:26.880811 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28551278-c831-45af-8fbe-a9c7155f1801-metrics-certs\") pod \"maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl\" (UID: \"28551278-c831-45af-8fbe-a9c7155f1801\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:29:26.896297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:26.896217 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc2ps\" (UniqueName: \"kubernetes.io/projected/28551278-c831-45af-8fbe-a9c7155f1801-kube-api-access-hc2ps\") pod \"maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl\" (UID: \"28551278-c831-45af-8fbe-a9c7155f1801\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:29:26.979680 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:26.978386 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:29:27.412196 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:27.412119 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl"] Oct 30 19:29:27.412399 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:29:27.412378 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28551278_c831_45af_8fbe_a9c7155f1801.slice/crio-cf77d9082345ebdaff2b1c46088b8297aba97942a8662f8f1de6b2ea222b8ebf WatchSource:0}: Error finding container cf77d9082345ebdaff2b1c46088b8297aba97942a8662f8f1de6b2ea222b8ebf: Status 404 returned error can't find the container with id cf77d9082345ebdaff2b1c46088b8297aba97942a8662f8f1de6b2ea222b8ebf Oct 30 19:29:27.810953 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:27.810920 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" event={"ID":"28551278-c831-45af-8fbe-a9c7155f1801","Type":"ContainerStarted","Data":"cf77d9082345ebdaff2b1c46088b8297aba97942a8662f8f1de6b2ea222b8ebf"} Oct 30 19:29:27.811321 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:27.810985 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:29:27.851717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:27.851651 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" podStartSLOduration=1.530524963 podStartE2EDuration="1.851634515s" totalImagesPullingTime="321.109552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:27.849226294 +0000 UTC m=+1520.923681654" watchObservedRunningTime="2026-10-30 19:29:27.851634515 +0000 UTC m=+1520.926089871" Oct 30 19:29:28.819342 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:28.819305 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" event={"ID":"28551278-c831-45af-8fbe-a9c7155f1801","Type":"ContainerStarted","Data":"47b817dd766513b68e93991e98853162b728d40dde1dd8fbd9d6d9f9114ee493"} Oct 30 19:29:31.625462 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.625418 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6"] Oct 30 19:29:31.647821 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.647789 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6"] Oct 30 19:29:31.648250 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.648229 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.706875 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.704430 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.706875 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.704659 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.733701 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.704718 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.733909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.733709 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcsfj\" (UniqueName: \"kubernetes.io/projected/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-kube-api-access-qcsfj\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.733909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.733760 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.733909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.733902 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.792909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.734122 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.792909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.734204 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.792909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.734324 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.859054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.858646 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.859054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.858755 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.859054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.858936 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.859054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.858990 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.869029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.859649 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcsfj\" (UniqueName: \"kubernetes.io/projected/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-kube-api-access-qcsfj\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.869029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.859731 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.869029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.859847 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.869029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.860508 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istiod-ca-cert\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.869029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.859589 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-credential-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.977448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.880366 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-workload-socket\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.977448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.880965 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-workload-certs\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.977448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.881088 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.977448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.881147 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.977448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.881780 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-data\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:31.977448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:31.921891 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-envoy\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:32.090769 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:32.090732 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-token\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:32.094993 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:32.094774 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-podinfo\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:32.097027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:32.096952 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcsfj\" (UniqueName: \"kubernetes.io/projected/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-kube-api-access-qcsfj\") pod \"e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:32.367676 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:32.367636 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:33.744037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:33.739730 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6"] Oct 30 19:29:34.050099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:34.049157 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:34.373726 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:34.373675 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:29:34.373861 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:34.373791 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:29:34.373861 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:34.373836 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:29:34.534936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:34.515801 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 30 19:29:34.716186 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:34.716131 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:35.053202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:35.053087 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:35.923807 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:35.906296 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:35.949950 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:35.949898 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 19:29:36.005129 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:36.005093 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:29:36.005450 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:36.005430 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:29:36.005646 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:36.005631 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:29:36.005813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:36.005800 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:29:36.017417 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:36.017337 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:29:36.526689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:36.518675 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:36.564337 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:36.533139 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:36.743046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:36.742982 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" event={"ID":"a4f73d85-9e6c-474c-907c-f1ae40bdcbda","Type":"ContainerStarted","Data":"5a18d6cb3e81fc47981e995056919b0a483017c7f512becda5f7c3a7163e3308"} Oct 30 19:29:36.743379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:36.743351 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" event={"ID":"a4f73d85-9e6c-474c-907c-f1ae40bdcbda","Type":"ContainerStarted","Data":"2529ed4c70061d703a9c4d565bcb9cc879a72f1f4251e38998993c5007aac56e"} Oct 30 19:29:36.803348 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:36.799958 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:29:36.972584 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:36.972522 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:37.032412 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:37.032366 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:29:37.070092 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:37.047369 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" podStartSLOduration=6.047348933 podStartE2EDuration="6.047348933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:36.903251958 +0000 UTC m=+1529.977707319" watchObservedRunningTime="2026-10-30 19:29:37.047348933 +0000 UTC m=+1530.121804293" Oct 30 19:29:37.385200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:37.385163 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:37.386278 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:37.386225 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:29:37.386406 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:37.386313 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" podUID="a4f73d85-9e6c-474c-907c-f1ae40bdcbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:29:38.369384 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:38.369339 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:29:38.369906 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:38.369414 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" podUID="a4f73d85-9e6c-474c-907c-f1ae40bdcbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:29:39.369171 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:39.369134 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:29:39.369357 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:39.369194 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" podUID="a4f73d85-9e6c-474c-907c-f1ae40bdcbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:29:39.602773 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:29:39.602731 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:29:40.368768 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:40.368732 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:29:40.368980 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:40.368791 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" podUID="a4f73d85-9e6c-474c-907c-f1ae40bdcbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:29:41.368589 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:41.368547 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:29:41.369111 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:41.368626 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" podUID="a4f73d85-9e6c-474c-907c-f1ae40bdcbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:29:42.371376 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:42.371333 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:29:42.372056 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:42.372022 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" podUID="a4f73d85-9e6c-474c-907c-f1ae40bdcbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:29:43.368827 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:43.368789 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:29:43.369027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:43.368846 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" podUID="a4f73d85-9e6c-474c-907c-f1ae40bdcbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 19:29:44.373630 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:44.373596 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:44.374115 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:44.374045 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:44.374979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:44.374960 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:29:46.945550 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:46.945488 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb"] Oct 30 19:29:46.982239 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:46.982199 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx"] Oct 30 19:29:49.367957 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:49.367914 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54"] Oct 30 19:29:49.368414 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:49.368204 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" podUID="401ee1cc-c3eb-4564-8429-df95be765de8" containerName="maas-api" containerID="cri-o://2c570a086b3dc3720811ca4a22ceb5632d6832b3e7236ab1198f6e0328c9a3af" gracePeriod=30 Oct 30 19:29:49.912046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:49.904177 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:49.942633 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:49.942597 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/401ee1cc-c3eb-4564-8429-df95be765de8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/401ee1cc-c3eb-4564-8429-df95be765de8-metrics-certs\") pod \"401ee1cc-c3eb-4564-8429-df95be765de8\" (UID: \"401ee1cc-c3eb-4564-8429-df95be765de8\") " Oct 30 19:29:49.943080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:49.943055 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/401ee1cc-c3eb-4564-8429-df95be765de8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401ee1cc-c3eb-4564-8429-df95be765de8-openshift-service-ca\") pod \"401ee1cc-c3eb-4564-8429-df95be765de8\" (UID: \"401ee1cc-c3eb-4564-8429-df95be765de8\") " Oct 30 19:29:50.008318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.006067 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/401ee1cc-c3eb-4564-8429-df95be765de8-kube-api-access-ctn9b\" (UniqueName: \"kubernetes.io/projected/401ee1cc-c3eb-4564-8429-df95be765de8-kube-api-access-ctn9b\") pod \"401ee1cc-c3eb-4564-8429-df95be765de8\" (UID: \"401ee1cc-c3eb-4564-8429-df95be765de8\") " Oct 30 19:29:50.008318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.006401 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/401ee1cc-c3eb-4564-8429-df95be765de8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401ee1cc-c3eb-4564-8429-df95be765de8-maas-api-tls\") pod \"401ee1cc-c3eb-4564-8429-df95be765de8\" (UID: \"401ee1cc-c3eb-4564-8429-df95be765de8\") " Oct 30 19:29:50.011418 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:49.946737 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401ee1cc-c3eb-4564-8429-df95be765de8-openshift-service-ca" pod "401ee1cc-c3eb-4564-8429-df95be765de8" (UID: "401ee1cc-c3eb-4564-8429-df95be765de8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:50.104324 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.104275 5047 generic.go:356] "Generic (PLEG): container finished" podID="401ee1cc-c3eb-4564-8429-df95be765de8" containerID="2c570a086b3dc3720811ca4a22ceb5632d6832b3e7236ab1198f6e0328c9a3af" exitCode=0 Oct 30 19:29:50.104855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.104818 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" event={"ID":"401ee1cc-c3eb-4564-8429-df95be765de8","Type":"ContainerDied","Data":"2c570a086b3dc3720811ca4a22ceb5632d6832b3e7236ab1198f6e0328c9a3af"} Oct 30 19:29:50.105066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.105039 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" event={"ID":"401ee1cc-c3eb-4564-8429-df95be765de8","Type":"ContainerDied","Data":"c4e0baa769a5b186ba7ea7074c0fe5285c86df894ae757e3f2f2c48109bde26d"} Oct 30 19:29:50.105313 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.105288 5047 scope.go:122] "RemoveContainer" containerID="2c570a086b3dc3720811ca4a22ceb5632d6832b3e7236ab1198f6e0328c9a3af" Oct 30 19:29:50.105890 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.105863 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54" Oct 30 19:29:50.124114 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.124062 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401ee1cc-c3eb-4564-8429-df95be765de8-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:29:50.149727 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.149685 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/401ee1cc-c3eb-4564-8429-df95be765de8-metrics-certs" pod "401ee1cc-c3eb-4564-8429-df95be765de8" (UID: "401ee1cc-c3eb-4564-8429-df95be765de8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:29:50.160041 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.156527 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401ee1cc-c3eb-4564-8429-df95be765de8-kube-api-access-ctn9b" pod "401ee1cc-c3eb-4564-8429-df95be765de8" (UID: "401ee1cc-c3eb-4564-8429-df95be765de8"). InnerVolumeSpecName "kube-api-access-ctn9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:50.160313 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.160055 5047 scope.go:122] "RemoveContainer" containerID="2c570a086b3dc3720811ca4a22ceb5632d6832b3e7236ab1198f6e0328c9a3af" Oct 30 19:29:50.178085 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:29:50.160799 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c570a086b3dc3720811ca4a22ceb5632d6832b3e7236ab1198f6e0328c9a3af\": container with ID starting with 2c570a086b3dc3720811ca4a22ceb5632d6832b3e7236ab1198f6e0328c9a3af not found: ID does not exist" containerID="2c570a086b3dc3720811ca4a22ceb5632d6832b3e7236ab1198f6e0328c9a3af" Oct 30 19:29:50.178085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.160846 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c570a086b3dc3720811ca4a22ceb5632d6832b3e7236ab1198f6e0328c9a3af"} err="failed to get container status \"2c570a086b3dc3720811ca4a22ceb5632d6832b3e7236ab1198f6e0328c9a3af\": rpc error: code = NotFound desc = could not find container \"2c570a086b3dc3720811ca4a22ceb5632d6832b3e7236ab1198f6e0328c9a3af\": container with ID starting with 2c570a086b3dc3720811ca4a22ceb5632d6832b3e7236ab1198f6e0328c9a3af not found: ID does not exist" Oct 30 19:29:50.178085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.172616 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/401ee1cc-c3eb-4564-8429-df95be765de8-maas-api-tls" pod "401ee1cc-c3eb-4564-8429-df95be765de8" (UID: "401ee1cc-c3eb-4564-8429-df95be765de8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:29:50.252035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.241532 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctn9b\" (UniqueName: \"kubernetes.io/projected/401ee1cc-c3eb-4564-8429-df95be765de8-kube-api-access-ctn9b\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:29:50.252035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.241569 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401ee1cc-c3eb-4564-8429-df95be765de8-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:29:50.252035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.241584 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/401ee1cc-c3eb-4564-8429-df95be765de8-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:29:50.571035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.567053 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54"] Oct 30 19:29:50.582164 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:50.582125 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d4245aa3-7c7fd48bd4-slf54"] Oct 30 19:29:51.335027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:51.326419 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="401ee1cc-c3eb-4564-8429-df95be765de8" path="/var/lib/kubelet/pods/401ee1cc-c3eb-4564-8429-df95be765de8/volumes" Oct 30 19:29:51.976030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:51.948081 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" podUID="c9584986-cf2e-47b2-af7a-16edc82d769a" containerName="payload-processing" containerID="cri-o://61144f283904c689716a5b406bdc75f2b94f6fcbcf5e4bb5f4a49464af3b1b38" gracePeriod=25 Oct 30 19:29:52.035949 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:51.995830 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" podUID="d6a1ea21-bd4e-4f7b-8d35-737204ee704d" containerName="payload-pre-processing" containerID="cri-o://a05107fd86f7acbcf39bc42a7e0fbd275ff58ca315c84d1fef764441b5194946" gracePeriod=25 Oct 30 19:29:52.430028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:52.429299 5047 generic.go:356] "Generic (PLEG): container finished" podID="c9584986-cf2e-47b2-af7a-16edc82d769a" containerID="61144f283904c689716a5b406bdc75f2b94f6fcbcf5e4bb5f4a49464af3b1b38" exitCode=0 Oct 30 19:29:52.430028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:52.429406 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" event={"ID":"c9584986-cf2e-47b2-af7a-16edc82d769a","Type":"ContainerDied","Data":"61144f283904c689716a5b406bdc75f2b94f6fcbcf5e4bb5f4a49464af3b1b38"} Oct 30 19:29:52.450030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:52.431718 5047 generic.go:356] "Generic (PLEG): container finished" podID="d6a1ea21-bd4e-4f7b-8d35-737204ee704d" containerID="a05107fd86f7acbcf39bc42a7e0fbd275ff58ca315c84d1fef764441b5194946" exitCode=0 Oct 30 19:29:52.450030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:52.431768 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" event={"ID":"d6a1ea21-bd4e-4f7b-8d35-737204ee704d","Type":"ContainerDied","Data":"a05107fd86f7acbcf39bc42a7e0fbd275ff58ca315c84d1fef764441b5194946"} Oct 30 19:29:52.513028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:52.506859 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx"] Oct 30 19:29:52.513028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:52.507995 5047 cpu_manager.go:414] "containerMap: removing container" podUID="401ee1cc-c3eb-4564-8429-df95be765de8" containerName="maas-api" Oct 30 19:29:52.513028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:52.508039 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401ee1cc-c3eb-4564-8429-df95be765de8" containerName="maas-api" Oct 30 19:29:52.513028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:52.508302 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="401ee1cc-c3eb-4564-8429-df95be765de8" containerName="maas-api" Oct 30 19:29:52.849034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:52.848488 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:52.933032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:52.894350 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f215f8b0-gw-openshift-default-dockercfg-l9n67\"" Oct 30 19:29:52.933032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:52.921658 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:52.967029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:52.935888 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx"] Oct 30 19:29:53.048033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:52.965379 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:53.073670 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:52.983106 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:53.074645 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.024804 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:53.248088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.234106 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-podinfo\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.248088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.234227 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-workload-socket\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.248088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.234276 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-token\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.248088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.234356 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istiod-ca-cert\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.248088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.234526 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-credential-socket\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.248088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.234589 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xx6z\" (UniqueName: \"kubernetes.io/projected/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-kube-api-access-9xx6z\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.248088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.234668 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-workload-certs\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.248088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.234734 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-envoy\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.248088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.234833 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-data\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.371033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.360392 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istiod-ca-cert\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.371033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.360567 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-credential-socket\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.371033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.360629 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xx6z\" (UniqueName: \"kubernetes.io/projected/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-kube-api-access-9xx6z\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.371033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.360700 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-workload-certs\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.371033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.360754 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-envoy\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.371033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.360839 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-data\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.371033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.360892 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-podinfo\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.371033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.360974 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-workload-socket\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.371033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.361035 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-token\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.428043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.425534 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istiod-ca-cert\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.428043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.425868 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-credential-socket\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.464320 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.461584 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-data\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.464320 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.461869 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-workload-certs\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.497948 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.497907 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-workload-socket\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.547113 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.529515 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-envoy\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.688158 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.688106 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-podinfo\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.774067 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.757030 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-token\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:53.774067 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:53.758803 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xx6z\" (UniqueName: \"kubernetes.io/projected/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-kube-api-access-9xx6z\") pod \"e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:54.235106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.235068 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:54.349575 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:29:54.349489 5047 logging.go:55] [core] [Channel #736 SubChannel #737]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.91:50052", ServerName: "10.132.0.91:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.91:50052: connect: connection refused" Oct 30 19:29:54.574114 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.574070 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" Oct 30 19:29:54.574375 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.574341 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" event={"ID":"d6a1ea21-bd4e-4f7b-8d35-737204ee704d","Type":"ContainerDied","Data":"c7aa074e4c69ad0b7bba11427787e30e237628b1eb464ccf473a146980867cbf"} Oct 30 19:29:54.574458 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.574414 5047 scope.go:122] "RemoveContainer" containerID="a05107fd86f7acbcf39bc42a7e0fbd275ff58ca315c84d1fef764441b5194946" Oct 30 19:29:54.636259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.636183 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a1ea21-bd4e-4f7b-8d35-737204ee704d-kube-api-access-k746v\" (UniqueName: \"kubernetes.io/projected/d6a1ea21-bd4e-4f7b-8d35-737204ee704d-kube-api-access-k746v\") pod \"d6a1ea21-bd4e-4f7b-8d35-737204ee704d\" (UID: \"d6a1ea21-bd4e-4f7b-8d35-737204ee704d\") " Oct 30 19:29:54.637484 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.636546 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6a1ea21-bd4e-4f7b-8d35-737204ee704d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a1ea21-bd4e-4f7b-8d35-737204ee704d-plugins-config-volume\") pod \"d6a1ea21-bd4e-4f7b-8d35-737204ee704d\" (UID: \"d6a1ea21-bd4e-4f7b-8d35-737204ee704d\") " Oct 30 19:29:54.637484 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.637315 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a1ea21-bd4e-4f7b-8d35-737204ee704d-plugins-config-volume" pod "d6a1ea21-bd4e-4f7b-8d35-737204ee704d" (UID: "d6a1ea21-bd4e-4f7b-8d35-737204ee704d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:54.637484 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.637449 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a1ea21-bd4e-4f7b-8d35-737204ee704d-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:29:54.663470 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.657684 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a1ea21-bd4e-4f7b-8d35-737204ee704d-kube-api-access-k746v" pod "d6a1ea21-bd4e-4f7b-8d35-737204ee704d" (UID: "d6a1ea21-bd4e-4f7b-8d35-737204ee704d"). InnerVolumeSpecName "kube-api-access-k746v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:54.739290 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.739261 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k746v\" (UniqueName: \"kubernetes.io/projected/d6a1ea21-bd4e-4f7b-8d35-737204ee704d-kube-api-access-k746v\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:29:54.783539 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.783497 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" Oct 30 19:29:54.830049 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.829978 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx"] Oct 30 19:29:54.833861 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:29:54.833834 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b43cbc_ec7e_4388_ba6f_9523c56d33b9.slice/crio-85bd3cfb180b0ab8b198d254d943e1157e0e9d5ec8206b69c5abedbf97432a73 WatchSource:0}: Error finding container 85bd3cfb180b0ab8b198d254d943e1157e0e9d5ec8206b69c5abedbf97432a73: Status 404 returned error can't find the container with id 85bd3cfb180b0ab8b198d254d943e1157e0e9d5ec8206b69c5abedbf97432a73 Oct 30 19:29:54.837245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.837213 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:29:54.837361 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.837315 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:29:54.837419 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.837364 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:29:54.839755 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.839731 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9584986-cf2e-47b2-af7a-16edc82d769a-kube-api-access-qnjgc\" (UniqueName: \"kubernetes.io/projected/c9584986-cf2e-47b2-af7a-16edc82d769a-kube-api-access-qnjgc\") pod \"c9584986-cf2e-47b2-af7a-16edc82d769a\" (UID: \"c9584986-cf2e-47b2-af7a-16edc82d769a\") " Oct 30 19:29:54.840048 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.839891 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9584986-cf2e-47b2-af7a-16edc82d769a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9584986-cf2e-47b2-af7a-16edc82d769a-plugins-config-volume\") pod \"c9584986-cf2e-47b2-af7a-16edc82d769a\" (UID: \"c9584986-cf2e-47b2-af7a-16edc82d769a\") " Oct 30 19:29:54.842176 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.842145 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9584986-cf2e-47b2-af7a-16edc82d769a-kube-api-access-qnjgc" pod "c9584986-cf2e-47b2-af7a-16edc82d769a" (UID: "c9584986-cf2e-47b2-af7a-16edc82d769a"). InnerVolumeSpecName "kube-api-access-qnjgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:54.843113 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.843088 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9584986-cf2e-47b2-af7a-16edc82d769a-plugins-config-volume" pod "c9584986-cf2e-47b2-af7a-16edc82d769a" (UID: "c9584986-cf2e-47b2-af7a-16edc82d769a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:54.960124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.960081 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnjgc\" (UniqueName: \"kubernetes.io/projected/c9584986-cf2e-47b2-af7a-16edc82d769a-kube-api-access-qnjgc\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:29:54.960124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:54.960119 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9584986-cf2e-47b2-af7a-16edc82d769a-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:29:55.283531 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:55.283432 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" podUID="c9584986-cf2e-47b2-af7a-16edc82d769a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.91:50052\" within 1s: context deadline exceeded" Oct 30 19:29:55.307339 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:55.307294 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" podUID="d6a1ea21-bd4e-4f7b-8d35-737204ee704d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.90:50052\" within 1s: context deadline exceeded" Oct 30 19:29:55.307508 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:29:55.307419 5047 logging.go:55] [core] [Channel #734 SubChannel #735]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.90:50052", ServerName: "10.132.0.90:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.90:50052: operation was canceled" Oct 30 19:29:55.583495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:55.583466 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" Oct 30 19:29:55.583665 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:55.583467 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb" event={"ID":"c9584986-cf2e-47b2-af7a-16edc82d769a","Type":"ContainerDied","Data":"d8c19c4803afbda30ad5dbd4fbfc8060b14e957c5a6fef5cf5f810eb4fe670e5"} Oct 30 19:29:55.583665 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:55.583598 5047 scope.go:122] "RemoveContainer" containerID="61144f283904c689716a5b406bdc75f2b94f6fcbcf5e4bb5f4a49464af3b1b38" Oct 30 19:29:55.585206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:55.584588 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx" Oct 30 19:29:55.586986 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:55.586955 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" event={"ID":"73b43cbc-ec7e-4388-ba6f-9523c56d33b9","Type":"ContainerStarted","Data":"d2491b1a0e4e45bc332c4f5efd18e50f9edad8a5a2731c007707b0dea4801366"} Oct 30 19:29:55.587148 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:55.587033 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" event={"ID":"73b43cbc-ec7e-4388-ba6f-9523c56d33b9","Type":"ContainerStarted","Data":"85bd3cfb180b0ab8b198d254d943e1157e0e9d5ec8206b69c5abedbf97432a73"} Oct 30 19:29:55.648215 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:55.648159 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" podStartSLOduration=3.648143659 podStartE2EDuration="3.648143659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:55.645693651 +0000 UTC m=+1548.720149009" watchObservedRunningTime="2026-10-30 19:29:55.648143659 +0000 UTC m=+1548.722599080" Oct 30 19:29:55.682877 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:55.682833 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx"] Oct 30 19:29:55.688175 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:55.688148 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d4245aa3-75cd9cf587-88jwx"] Oct 30 19:29:55.723124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:55.723089 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb"] Oct 30 19:29:55.726766 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:55.726738 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d4245aa3-5ffd98cfc8-wmpgb"] Oct 30 19:29:56.239736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:56.239706 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:56.245413 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:56.245384 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:56.595822 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:56.595792 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:56.596744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:56.596725 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:29:57.054149 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:57.054076 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9584986-cf2e-47b2-af7a-16edc82d769a" path="/var/lib/kubelet/pods/c9584986-cf2e-47b2-af7a-16edc82d769a/volumes" Oct 30 19:29:57.054524 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:57.054509 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6a1ea21-bd4e-4f7b-8d35-737204ee704d" path="/var/lib/kubelet/pods/d6a1ea21-bd4e-4f7b-8d35-737204ee704d/volumes" Oct 30 19:29:58.854589 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:58.772469 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb"] Oct 30 19:29:58.892152 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:58.892120 5047 cpu_manager.go:414] "containerMap: removing container" podUID="c9584986-cf2e-47b2-af7a-16edc82d769a" containerName="payload-processing" Oct 30 19:29:58.892374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:58.892357 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9584986-cf2e-47b2-af7a-16edc82d769a" containerName="payload-processing" Oct 30 19:29:58.892580 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:58.892566 5047 cpu_manager.go:414] "containerMap: removing container" podUID="d6a1ea21-bd4e-4f7b-8d35-737204ee704d" containerName="payload-pre-processing" Oct 30 19:29:58.892660 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:58.892648 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a1ea21-bd4e-4f7b-8d35-737204ee704d" containerName="payload-pre-processing" Oct 30 19:29:58.892913 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:58.892898 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a1ea21-bd4e-4f7b-8d35-737204ee704d" containerName="payload-pre-processing" Oct 30 19:29:58.893028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:58.893016 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9584986-cf2e-47b2-af7a-16edc82d769a" containerName="payload-processing" Oct 30 19:29:58.904174 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:58.904146 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" Oct 30 19:29:58.928026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:58.923169 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb"] Oct 30 19:29:58.971338 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:58.971307 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f215f8b0-dockercfg-787vr\"" Oct 30 19:29:58.972314 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:58.971901 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f215f8b0\"" Oct 30 19:29:59.094628 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:59.094366 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d20bf7a-be4f-48b3-b9b0-d39ca9320150-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb\" (UID: \"4d20bf7a-be4f-48b3-b9b0-d39ca9320150\") " pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" Oct 30 19:29:59.094628 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:59.094436 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qkj4\" (UniqueName: \"kubernetes.io/projected/4d20bf7a-be4f-48b3-b9b0-d39ca9320150-kube-api-access-6qkj4\") pod \"payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb\" (UID: \"4d20bf7a-be4f-48b3-b9b0-d39ca9320150\") " pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" Oct 30 19:29:59.196520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:59.196428 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d20bf7a-be4f-48b3-b9b0-d39ca9320150-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb\" (UID: \"4d20bf7a-be4f-48b3-b9b0-d39ca9320150\") " pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" Oct 30 19:29:59.196520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:59.196486 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qkj4\" (UniqueName: \"kubernetes.io/projected/4d20bf7a-be4f-48b3-b9b0-d39ca9320150-kube-api-access-6qkj4\") pod \"payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb\" (UID: \"4d20bf7a-be4f-48b3-b9b0-d39ca9320150\") " pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" Oct 30 19:29:59.197301 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:59.197280 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d20bf7a-be4f-48b3-b9b0-d39ca9320150-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb\" (UID: \"4d20bf7a-be4f-48b3-b9b0-d39ca9320150\") " pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" Oct 30 19:29:59.219603 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:59.219563 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qkj4\" (UniqueName: \"kubernetes.io/projected/4d20bf7a-be4f-48b3-b9b0-d39ca9320150-kube-api-access-6qkj4\") pod \"payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb\" (UID: \"4d20bf7a-be4f-48b3-b9b0-d39ca9320150\") " pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" Oct 30 19:29:59.424966 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:59.424925 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" Oct 30 19:29:59.789233 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:29:59.786695 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb"] Oct 30 19:30:00.645994 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:00.645962 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" event={"ID":"4d20bf7a-be4f-48b3-b9b0-d39ca9320150","Type":"ContainerStarted","Data":"6ef430a95fb48e42af16563c46f2cad2fc0e9d7292e3084271854fbbd96912f0"} Oct 30 19:30:00.646580 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:00.646561 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" Oct 30 19:30:00.646674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:00.646662 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" event={"ID":"4d20bf7a-be4f-48b3-b9b0-d39ca9320150","Type":"ContainerStarted","Data":"29de9a2273791d4cb5524a5d24c6611cb523305a6fe4b0a7fe0f5e886bc336bd"} Oct 30 19:30:00.684795 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:00.684732 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" podStartSLOduration=2.684717429 podStartE2EDuration="2.684717429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:00.683484797 +0000 UTC m=+1553.757940157" watchObservedRunningTime="2026-10-30 19:30:00.684717429 +0000 UTC m=+1553.759172786" Oct 30 19:30:04.649703 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:04.649663 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2"] Oct 30 19:30:07.252234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.252195 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl"] Oct 30 19:30:07.252759 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.252509 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" podUID="28551278-c831-45af-8fbe-a9c7155f1801" containerName="maas-api" containerID="cri-o://47b817dd766513b68e93991e98853162b728d40dde1dd8fbd9d6d9f9114ee493" gracePeriod=30 Oct 30 19:30:07.782654 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.782563 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:30:07.786198 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.786148 5047 generic.go:356] "Generic (PLEG): container finished" podID="28551278-c831-45af-8fbe-a9c7155f1801" containerID="47b817dd766513b68e93991e98853162b728d40dde1dd8fbd9d6d9f9114ee493" exitCode=0 Oct 30 19:30:07.786611 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.786586 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" event={"ID":"28551278-c831-45af-8fbe-a9c7155f1801","Type":"ContainerDied","Data":"47b817dd766513b68e93991e98853162b728d40dde1dd8fbd9d6d9f9114ee493"} Oct 30 19:30:07.786871 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.786852 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" event={"ID":"28551278-c831-45af-8fbe-a9c7155f1801","Type":"ContainerDied","Data":"cf77d9082345ebdaff2b1c46088b8297aba97942a8662f8f1de6b2ea222b8ebf"} Oct 30 19:30:07.793161 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.793127 5047 scope.go:122] "RemoveContainer" containerID="47b817dd766513b68e93991e98853162b728d40dde1dd8fbd9d6d9f9114ee493" Oct 30 19:30:07.804688 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.804646 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28551278-c831-45af-8fbe-a9c7155f1801-kube-api-access-hc2ps\" (UniqueName: \"kubernetes.io/projected/28551278-c831-45af-8fbe-a9c7155f1801-kube-api-access-hc2ps\") pod \"28551278-c831-45af-8fbe-a9c7155f1801\" (UID: \"28551278-c831-45af-8fbe-a9c7155f1801\") " Oct 30 19:30:07.805622 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.805582 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28551278-c831-45af-8fbe-a9c7155f1801-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28551278-c831-45af-8fbe-a9c7155f1801-metrics-certs\") pod \"28551278-c831-45af-8fbe-a9c7155f1801\" (UID: \"28551278-c831-45af-8fbe-a9c7155f1801\") " Oct 30 19:30:07.806259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.806240 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28551278-c831-45af-8fbe-a9c7155f1801-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28551278-c831-45af-8fbe-a9c7155f1801-maas-api-tls\") pod \"28551278-c831-45af-8fbe-a9c7155f1801\" (UID: \"28551278-c831-45af-8fbe-a9c7155f1801\") " Oct 30 19:30:07.806825 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.806805 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28551278-c831-45af-8fbe-a9c7155f1801-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28551278-c831-45af-8fbe-a9c7155f1801-openshift-service-ca\") pod \"28551278-c831-45af-8fbe-a9c7155f1801\" (UID: \"28551278-c831-45af-8fbe-a9c7155f1801\") " Oct 30 19:30:07.810072 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.807311 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28551278-c831-45af-8fbe-a9c7155f1801-openshift-service-ca" pod "28551278-c831-45af-8fbe-a9c7155f1801" (UID: "28551278-c831-45af-8fbe-a9c7155f1801"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:07.863634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.863605 5047 scope.go:122] "RemoveContainer" containerID="47b817dd766513b68e93991e98853162b728d40dde1dd8fbd9d6d9f9114ee493" Oct 30 19:30:07.866769 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:30:07.866693 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47b817dd766513b68e93991e98853162b728d40dde1dd8fbd9d6d9f9114ee493\": container with ID starting with 47b817dd766513b68e93991e98853162b728d40dde1dd8fbd9d6d9f9114ee493 not found: ID does not exist" containerID="47b817dd766513b68e93991e98853162b728d40dde1dd8fbd9d6d9f9114ee493" Oct 30 19:30:07.866769 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.866738 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47b817dd766513b68e93991e98853162b728d40dde1dd8fbd9d6d9f9114ee493"} err="failed to get container status \"47b817dd766513b68e93991e98853162b728d40dde1dd8fbd9d6d9f9114ee493\": rpc error: code = NotFound desc = could not find container \"47b817dd766513b68e93991e98853162b728d40dde1dd8fbd9d6d9f9114ee493\": container with ID starting with 47b817dd766513b68e93991e98853162b728d40dde1dd8fbd9d6d9f9114ee493 not found: ID does not exist" Oct 30 19:30:07.906947 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.906703 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28551278-c831-45af-8fbe-a9c7155f1801-maas-api-tls" pod "28551278-c831-45af-8fbe-a9c7155f1801" (UID: "28551278-c831-45af-8fbe-a9c7155f1801"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:07.906947 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.906810 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28551278-c831-45af-8fbe-a9c7155f1801-metrics-certs" pod "28551278-c831-45af-8fbe-a9c7155f1801" (UID: "28551278-c831-45af-8fbe-a9c7155f1801"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:07.906947 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.906920 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28551278-c831-45af-8fbe-a9c7155f1801-kube-api-access-hc2ps" pod "28551278-c831-45af-8fbe-a9c7155f1801" (UID: "28551278-c831-45af-8fbe-a9c7155f1801"). InnerVolumeSpecName "kube-api-access-hc2ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:07.912573 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.912466 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28551278-c831-45af-8fbe-a9c7155f1801-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:30:07.912573 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.912511 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28551278-c831-45af-8fbe-a9c7155f1801-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:30:07.912573 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.912528 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc2ps\" (UniqueName: \"kubernetes.io/projected/28551278-c831-45af-8fbe-a9c7155f1801-kube-api-access-hc2ps\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:30:07.912573 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:07.912543 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28551278-c831-45af-8fbe-a9c7155f1801-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:30:08.817015 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:08.816966 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl" Oct 30 19:30:08.903899 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:08.903862 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl"] Oct 30 19:30:08.925657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:08.925624 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c5e08e43-775f6b655f-4qrwl"] Oct 30 19:30:09.051743 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:09.051704 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28551278-c831-45af-8fbe-a9c7155f1801" path="/var/lib/kubelet/pods/28551278-c831-45af-8fbe-a9c7155f1801/volumes" Oct 30 19:30:09.605037 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:30:09.604966 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:30:09.650283 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:09.650240 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" podUID="d066642d-6752-4173-859b-2ed57ecbba40" containerName="payload-pre-processing" containerID="cri-o://a7b6a77688cec6cff6be8b55b98e2e9ce175f590846752c7914de7aa9929dbd7" gracePeriod=25 Oct 30 19:30:09.826590 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:09.826556 5047 generic.go:356] "Generic (PLEG): container finished" podID="d066642d-6752-4173-859b-2ed57ecbba40" containerID="a7b6a77688cec6cff6be8b55b98e2e9ce175f590846752c7914de7aa9929dbd7" exitCode=0 Oct 30 19:30:09.827128 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:09.826602 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" event={"ID":"d066642d-6752-4173-859b-2ed57ecbba40","Type":"ContainerDied","Data":"a7b6a77688cec6cff6be8b55b98e2e9ce175f590846752c7914de7aa9929dbd7"} Oct 30 19:30:09.948800 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:09.948764 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" Oct 30 19:30:10.032189 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:10.032154 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d066642d-6752-4173-859b-2ed57ecbba40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d066642d-6752-4173-859b-2ed57ecbba40-plugins-config-volume\") pod \"d066642d-6752-4173-859b-2ed57ecbba40\" (UID: \"d066642d-6752-4173-859b-2ed57ecbba40\") " Oct 30 19:30:10.032393 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:10.032208 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d066642d-6752-4173-859b-2ed57ecbba40-kube-api-access-mwvls\" (UniqueName: \"kubernetes.io/projected/d066642d-6752-4173-859b-2ed57ecbba40-kube-api-access-mwvls\") pod \"d066642d-6752-4173-859b-2ed57ecbba40\" (UID: \"d066642d-6752-4173-859b-2ed57ecbba40\") " Oct 30 19:30:10.032676 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:10.032644 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d066642d-6752-4173-859b-2ed57ecbba40-plugins-config-volume" pod "d066642d-6752-4173-859b-2ed57ecbba40" (UID: "d066642d-6752-4173-859b-2ed57ecbba40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:10.033059 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:10.032990 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d066642d-6752-4173-859b-2ed57ecbba40-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:30:10.035848 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:10.035803 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d066642d-6752-4173-859b-2ed57ecbba40-kube-api-access-mwvls" pod "d066642d-6752-4173-859b-2ed57ecbba40" (UID: "d066642d-6752-4173-859b-2ed57ecbba40"). InnerVolumeSpecName "kube-api-access-mwvls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:10.194086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:10.186564 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwvls\" (UniqueName: \"kubernetes.io/projected/d066642d-6752-4173-859b-2ed57ecbba40-kube-api-access-mwvls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:30:10.987070 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:10.981195 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb"] Oct 30 19:30:10.987070 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:10.982991 5047 cpu_manager.go:414] "containerMap: removing container" podUID="d066642d-6752-4173-859b-2ed57ecbba40" containerName="payload-pre-processing" Oct 30 19:30:10.987070 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:10.983034 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d066642d-6752-4173-859b-2ed57ecbba40" containerName="payload-pre-processing" Oct 30 19:30:10.987070 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:10.983150 5047 cpu_manager.go:414] "containerMap: removing container" podUID="28551278-c831-45af-8fbe-a9c7155f1801" containerName="maas-api" Oct 30 19:30:10.987070 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:10.983160 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28551278-c831-45af-8fbe-a9c7155f1801" containerName="maas-api" Oct 30 19:30:10.987070 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:10.983530 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="28551278-c831-45af-8fbe-a9c7155f1801" containerName="maas-api" Oct 30 19:30:10.987070 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:10.983571 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="d066642d-6752-4173-859b-2ed57ecbba40" containerName="payload-pre-processing" Oct 30 19:30:11.212658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.212624 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb"] Oct 30 19:30:11.212960 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.212937 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" Oct 30 19:30:11.258271 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.250963 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" Oct 30 19:30:11.258271 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.253182 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-87b0dcce\"" Oct 30 19:30:11.277477 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.265288 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-87b0dcce-dockercfg-m8gk2\"" Oct 30 19:30:11.349029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.336732 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2" event={"ID":"d066642d-6752-4173-859b-2ed57ecbba40","Type":"ContainerDied","Data":"0ff68acf4427b7f2a35f994abdb0bc809833f3f479469bac4d89713e2647fa41"} Oct 30 19:30:11.359323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.359290 5047 scope.go:122] "RemoveContainer" containerID="a7b6a77688cec6cff6be8b55b98e2e9ce175f590846752c7914de7aa9929dbd7" Oct 30 19:30:11.359696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.350249 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qj95\" (UniqueName: \"kubernetes.io/projected/d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f-kube-api-access-5qj95\") pod \"payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb\" (UID: \"d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f\") " pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" Oct 30 19:30:11.389932 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.389680 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f-plugins-config-volume\") pod \"payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb\" (UID: \"d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f\") " pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" Oct 30 19:30:11.414050 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.411813 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq"] Oct 30 19:30:11.425955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.425260 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.509364 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.492412 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-istio-data\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.509364 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.492473 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-istio-envoy\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.509364 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.492602 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/592dcc49-fe49-4168-aa60-b51052026387-istio-token\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.509364 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.492666 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592dcc49-fe49-4168-aa60-b51052026387-istio-podinfo\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.509364 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.492740 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qj95\" (UniqueName: \"kubernetes.io/projected/d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f-kube-api-access-5qj95\") pod \"payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb\" (UID: \"d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f\") " pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" Oct 30 19:30:11.509364 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.492777 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t6h5\" (UniqueName: \"kubernetes.io/projected/592dcc49-fe49-4168-aa60-b51052026387-kube-api-access-4t6h5\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.509364 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.492805 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592dcc49-fe49-4168-aa60-b51052026387-istiod-ca-cert\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.509364 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.492941 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f-plugins-config-volume\") pod \"payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb\" (UID: \"d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f\") " pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" Oct 30 19:30:11.509364 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.492977 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-workload-socket\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.509364 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.493043 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-workload-certs\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.509364 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.504642 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3bdb4fd2-openshift-default-dockercfg-nhmnf\"" Oct 30 19:30:11.562266 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.543854 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq"] Oct 30 19:30:11.562266 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.548026 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f-plugins-config-volume\") pod \"payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb\" (UID: \"d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f\") " pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" Oct 30 19:30:11.562266 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.549478 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-credential-socket\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.628297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.611091 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qj95\" (UniqueName: \"kubernetes.io/projected/d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f-kube-api-access-5qj95\") pod \"payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb\" (UID: \"d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f\") " pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" Oct 30 19:30:11.690609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.650941 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t6h5\" (UniqueName: \"kubernetes.io/projected/592dcc49-fe49-4168-aa60-b51052026387-kube-api-access-4t6h5\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.690609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.651021 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592dcc49-fe49-4168-aa60-b51052026387-istiod-ca-cert\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.690609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.651149 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-workload-socket\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.690609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.651197 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-workload-certs\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.690609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.651256 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-credential-socket\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.690609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.651295 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-istio-data\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.690609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.651326 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-istio-envoy\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.690609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.651448 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/592dcc49-fe49-4168-aa60-b51052026387-istio-token\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.690609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.651498 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592dcc49-fe49-4168-aa60-b51052026387-istio-podinfo\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.690609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.653415 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-istio-data\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.690609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.653658 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-credential-socket\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.690609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.654305 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592dcc49-fe49-4168-aa60-b51052026387-istiod-ca-cert\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.690609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.655288 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-workload-socket\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.690609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.655904 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-workload-certs\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.690609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.677512 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592dcc49-fe49-4168-aa60-b51052026387-istio-podinfo\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.695199 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.690802 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-istio-envoy\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.703642 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.698278 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t6h5\" (UniqueName: \"kubernetes.io/projected/592dcc49-fe49-4168-aa60-b51052026387-kube-api-access-4t6h5\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.707838 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.704629 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/592dcc49-fe49-4168-aa60-b51052026387-istio-token\") pod \"e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.726038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.725076 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2"] Oct 30 19:30:11.750845 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.750772 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5e08e43-577c7df87f-swqv2"] Oct 30 19:30:11.815087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.814316 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:11.880951 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:11.880915 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" Oct 30 19:30:12.169283 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:12.169254 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq"] Oct 30 19:30:12.171780 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:30:12.171739 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592dcc49_fe49_4168_aa60_b51052026387.slice/crio-558c90378e010fa3f92ed0d0b113324aab2443927392a4d83fdfbdd025972113 WatchSource:0}: Error finding container 558c90378e010fa3f92ed0d0b113324aab2443927392a4d83fdfbdd025972113: Status 404 returned error can't find the container with id 558c90378e010fa3f92ed0d0b113324aab2443927392a4d83fdfbdd025972113 Oct 30 19:30:12.174889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:12.174856 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:30:12.174991 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:12.174927 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:30:12.174991 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:12.174958 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:30:12.210292 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:12.210264 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb"] Oct 30 19:30:12.215482 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:30:12.215455 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68f5e81_a8e6_4d59_9024_99f6a7f7ed0f.slice/crio-90e80a72f54e75eb9a715d61cc1d03a392346cd67f11e3f1f85618e01fa526f1 WatchSource:0}: Error finding container 90e80a72f54e75eb9a715d61cc1d03a392346cd67f11e3f1f85618e01fa526f1: Status 404 returned error can't find the container with id 90e80a72f54e75eb9a715d61cc1d03a392346cd67f11e3f1f85618e01fa526f1 Oct 30 19:30:12.263370 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:12.263341 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" event={"ID":"592dcc49-fe49-4168-aa60-b51052026387","Type":"ContainerStarted","Data":"558c90378e010fa3f92ed0d0b113324aab2443927392a4d83fdfbdd025972113"} Oct 30 19:30:12.265243 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:12.265213 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" event={"ID":"d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f","Type":"ContainerStarted","Data":"90e80a72f54e75eb9a715d61cc1d03a392346cd67f11e3f1f85618e01fa526f1"} Oct 30 19:30:12.663859 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:12.663826 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" Oct 30 19:30:13.051765 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:13.051682 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d066642d-6752-4173-859b-2ed57ecbba40" path="/var/lib/kubelet/pods/d066642d-6752-4173-859b-2ed57ecbba40/volumes" Oct 30 19:30:13.275418 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:13.275379 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" event={"ID":"d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f","Type":"ContainerStarted","Data":"b87972331e46c4a7a981ab7debf43e9b3808214824b56a2ca406b1611534b001"} Oct 30 19:30:13.275885 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:13.275460 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" Oct 30 19:30:13.278298 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:13.278269 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" event={"ID":"592dcc49-fe49-4168-aa60-b51052026387","Type":"ContainerStarted","Data":"e6e67279b91b253402119aa734fc4970f15cbbe113740ea65ec347659e66e7af"} Oct 30 19:30:13.317357 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:13.317252 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" podStartSLOduration=3.317233825 podStartE2EDuration="3.317233825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:13.313230329 +0000 UTC m=+1566.387685686" watchObservedRunningTime="2026-10-30 19:30:13.317233825 +0000 UTC m=+1566.391689184" Oct 30 19:30:13.357194 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:13.357145 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" podStartSLOduration=2.357129582 podStartE2EDuration="2.357129582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:13.352900636 +0000 UTC m=+1566.427355997" watchObservedRunningTime="2026-10-30 19:30:13.357129582 +0000 UTC m=+1566.431584938" Oct 30 19:30:13.815512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:13.815468 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:13.879183 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:13.879149 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:14.285935 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:14.285903 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:14.286744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:14.286722 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:30:17.086483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:17.086442 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb"] Oct 30 19:30:17.092292 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:17.092261 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" Oct 30 19:30:17.115629 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:17.115591 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb"] Oct 30 19:30:17.122246 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:17.122214 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3bdb4fd2\"" Oct 30 19:30:17.123933 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:17.123909 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3bdb4fd2-dockercfg-44bzd\"" Oct 30 19:30:17.226029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:17.225563 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xdwv\" (UniqueName: \"kubernetes.io/projected/c3098017-4e51-4b43-834d-6c2449565f6b-kube-api-access-4xdwv\") pod \"payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb\" (UID: \"c3098017-4e51-4b43-834d-6c2449565f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" Oct 30 19:30:17.226029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:17.225808 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3098017-4e51-4b43-834d-6c2449565f6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb\" (UID: \"c3098017-4e51-4b43-834d-6c2449565f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" Oct 30 19:30:17.336563 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:17.336460 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3098017-4e51-4b43-834d-6c2449565f6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb\" (UID: \"c3098017-4e51-4b43-834d-6c2449565f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" Oct 30 19:30:17.337337 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:17.337307 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xdwv\" (UniqueName: \"kubernetes.io/projected/c3098017-4e51-4b43-834d-6c2449565f6b-kube-api-access-4xdwv\") pod \"payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb\" (UID: \"c3098017-4e51-4b43-834d-6c2449565f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" Oct 30 19:30:17.347686 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:17.340220 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3098017-4e51-4b43-834d-6c2449565f6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb\" (UID: \"c3098017-4e51-4b43-834d-6c2449565f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" Oct 30 19:30:17.501033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:17.496950 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xdwv\" (UniqueName: \"kubernetes.io/projected/c3098017-4e51-4b43-834d-6c2449565f6b-kube-api-access-4xdwv\") pod \"payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb\" (UID: \"c3098017-4e51-4b43-834d-6c2449565f6b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" Oct 30 19:30:17.709551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:17.709444 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" Oct 30 19:30:18.023847 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:30:18.023795 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3098017_4e51_4b43_834d_6c2449565f6b.slice/crio-105c84e666541b4dcde026953c0888bb4626800fa3242fa9206e7ccc0b521316 WatchSource:0}: Error finding container 105c84e666541b4dcde026953c0888bb4626800fa3242fa9206e7ccc0b521316: Status 404 returned error can't find the container with id 105c84e666541b4dcde026953c0888bb4626800fa3242fa9206e7ccc0b521316 Oct 30 19:30:18.024765 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:18.024730 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb"] Oct 30 19:30:18.409267 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:18.409220 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" event={"ID":"c3098017-4e51-4b43-834d-6c2449565f6b","Type":"ContainerStarted","Data":"195613bc953a575153b1556cfbdf243b43c85b9356d234ec64c81b947ddd6f80"} Oct 30 19:30:18.409675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:18.409296 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" event={"ID":"c3098017-4e51-4b43-834d-6c2449565f6b","Type":"ContainerStarted","Data":"105c84e666541b4dcde026953c0888bb4626800fa3242fa9206e7ccc0b521316"} Oct 30 19:30:18.409675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:18.409326 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" Oct 30 19:30:18.451500 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:18.451445 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" podStartSLOduration=1.45142808 podStartE2EDuration="1.45142808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:18.45031503 +0000 UTC m=+1571.524770389" watchObservedRunningTime="2026-10-30 19:30:18.45142808 +0000 UTC m=+1571.525883438" Oct 30 19:30:24.289694 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:24.289656 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" Oct 30 19:30:26.239735 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:26.239699 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm"] Oct 30 19:30:26.244869 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:26.244842 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:30:26.258984 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:26.258954 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-87b0dcce-serving-cert\"" Oct 30 19:30:26.272921 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:26.272882 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm"] Oct 30 19:30:26.389512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:26.389476 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4kvl\" (UniqueName: \"kubernetes.io/projected/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-kube-api-access-b4kvl\") pod \"maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm\" (UID: \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:30:26.389691 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:26.389518 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-openshift-service-ca\") pod \"maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm\" (UID: \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:30:26.389763 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:26.389735 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-maas-api-tls\") pod \"maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm\" (UID: \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:30:26.389966 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:26.389944 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-metrics-certs\") pod \"maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm\" (UID: \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:30:26.496088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:26.493502 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-metrics-certs\") pod \"maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm\" (UID: \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:30:26.496088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:26.493625 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4kvl\" (UniqueName: \"kubernetes.io/projected/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-kube-api-access-b4kvl\") pod \"maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm\" (UID: \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:30:26.496088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:26.493655 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-openshift-service-ca\") pod \"maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm\" (UID: \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:30:26.496088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:26.493845 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-maas-api-tls\") pod \"maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm\" (UID: \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:30:26.496088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:26.495957 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-openshift-service-ca\") pod \"maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm\" (UID: \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:30:26.511231 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:26.500327 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-maas-api-tls\") pod \"maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm\" (UID: \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:30:26.527027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:26.523163 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4kvl\" (UniqueName: \"kubernetes.io/projected/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-kube-api-access-b4kvl\") pod \"maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm\" (UID: \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:30:26.531037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:26.528891 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-metrics-certs\") pod \"maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm\" (UID: \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:30:26.587029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:26.583607 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:30:27.027386 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:27.027349 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm"] Oct 30 19:30:27.524227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:27.524189 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" event={"ID":"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d","Type":"ContainerStarted","Data":"66dc7c192bbd5b2f7fc6abd79400c577683e4db7d06df7eb28dd6ea6464cc321"} Oct 30 19:30:27.524614 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:27.524253 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" event={"ID":"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d","Type":"ContainerStarted","Data":"c2cca957c00a45114f01aebcb5ad1b6d238ede12deb5af031782f053ad9f46f2"} Oct 30 19:30:27.524614 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:27.524279 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:30:27.568058 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:27.567980 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" podStartSLOduration=1.147274946 podStartE2EDuration="1.567961927s" totalImagesPullingTime="420.686981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:27.565705246 +0000 UTC m=+1580.640160604" watchObservedRunningTime="2026-10-30 19:30:27.567961927 +0000 UTC m=+1580.642417332" Oct 30 19:30:29.418821 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:29.418792 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" Oct 30 19:30:33.539900 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:33.539869 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:30:39.602555 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:30:39.602516 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:30:42.638255 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:42.638203 5047 patch_prober.go:29] interesting pod/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:30:42.638744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:42.638303 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" podUID="73b43cbc-ec7e-4388-ba6f-9523c56d33b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:44.809035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:44.808922 5047 generic.go:356] "Generic (PLEG): container finished" podID="73b43cbc-ec7e-4388-ba6f-9523c56d33b9" containerID="d2491b1a0e4e45bc332c4f5efd18e50f9edad8a5a2731c007707b0dea4801366" exitCode=0 Oct 30 19:30:44.809580 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:44.809047 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" event={"ID":"73b43cbc-ec7e-4388-ba6f-9523c56d33b9","Type":"ContainerDied","Data":"d2491b1a0e4e45bc332c4f5efd18e50f9edad8a5a2731c007707b0dea4801366"} Oct 30 19:30:44.809703 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:44.809683 5047 scope.go:122] "RemoveContainer" containerID="d2491b1a0e4e45bc332c4f5efd18e50f9edad8a5a2731c007707b0dea4801366" Oct 30 19:30:44.811539 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:44.811503 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:30:44.811650 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:44.811586 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:30:44.811650 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:44.811620 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:30:45.818876 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:45.818841 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" event={"ID":"73b43cbc-ec7e-4388-ba6f-9523c56d33b9","Type":"ContainerStarted","Data":"0dfb2bffac4021b489b749b1666862ca62645ff98b43f372c9136dcff4e8bbb0"} Oct 30 19:30:46.248027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:46.241436 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:30:46.248027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:46.241579 5047 patch_prober.go:29] interesting pod/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:30:46.248027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:46.241644 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" podUID="73b43cbc-ec7e-4388-ba6f-9523c56d33b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 19:30:47.247038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:47.246485 5047 patch_prober.go:29] interesting pod/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:30:47.247038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:47.246568 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" podUID="73b43cbc-ec7e-4388-ba6f-9523c56d33b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 19:30:48.246654 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:48.246609 5047 patch_prober.go:29] interesting pod/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:30:48.247554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:48.246949 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" podUID="73b43cbc-ec7e-4388-ba6f-9523c56d33b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 19:30:48.304733 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:48.304686 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb"] Oct 30 19:30:49.257504 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:49.257469 5047 patch_prober.go:29] interesting pod/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:30:49.258643 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:49.258235 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" podUID="73b43cbc-ec7e-4388-ba6f-9523c56d33b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 19:30:50.268038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.267185 5047 patch_prober.go:29] interesting pod/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:30:50.268038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.267274 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" podUID="73b43cbc-ec7e-4388-ba6f-9523c56d33b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 19:30:50.586081 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.580447 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z"] Oct 30 19:30:50.590417 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.590383 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 19:30:50.603671 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.603096 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z"] Oct 30 19:30:50.604044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.603975 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc929\" (UniqueName: \"kubernetes.io/projected/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-kube-api-access-tc929\") pod \"maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z\" (UID: \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 19:30:50.604614 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.604585 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z\" (UID: \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 19:30:50.604859 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.604836 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z\" (UID: \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 19:30:50.605089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.605070 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z\" (UID: \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 19:30:50.607418 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.607382 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-a56cb5-serving-cert\"" Oct 30 19:30:50.706223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.706188 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc929\" (UniqueName: \"kubernetes.io/projected/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-kube-api-access-tc929\") pod \"maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z\" (UID: \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 19:30:50.706425 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.706392 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z\" (UID: \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 19:30:50.706526 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.706491 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z\" (UID: \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 19:30:50.706600 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.706582 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z\" (UID: \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 19:30:50.707323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.707295 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z\" (UID: \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 19:30:50.709550 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.709525 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z\" (UID: \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 19:30:50.710884 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.710844 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z\" (UID: \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 19:30:50.727929 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.727903 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc929\" (UniqueName: \"kubernetes.io/projected/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-kube-api-access-tc929\") pod \"maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z\" (UID: \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 19:30:50.918442 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:50.918343 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 19:30:51.297495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:51.297458 5047 patch_prober.go:29] interesting pod/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:30:51.297973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:51.297518 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" podUID="73b43cbc-ec7e-4388-ba6f-9523c56d33b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 19:30:51.500195 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:51.500161 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z"] Oct 30 19:30:52.574372 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:52.570544 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" event={"ID":"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4","Type":"ContainerStarted","Data":"f430896edb3042f83a988d3c6afabe591e5c531f17f4b61aeb6befe85411fd6c"} Oct 30 19:30:52.574372 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:52.570619 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" event={"ID":"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4","Type":"ContainerStarted","Data":"c17a3bc28ea7afe9126f1d3968c7391bcff2d3820cb56e5dd023bc8108569589"} Oct 30 19:30:52.574372 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:52.572187 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 19:30:52.646139 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:52.646065 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" podStartSLOduration=2.216889612 podStartE2EDuration="2.646038769s" totalImagesPullingTime="429.149157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:52.643366429 +0000 UTC m=+1605.717821789" watchObservedRunningTime="2026-10-30 19:30:52.646038769 +0000 UTC m=+1605.720494132" Oct 30 19:30:52.801349 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:52.801303 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:30:52.803200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:52.803166 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:30:52.833310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:52.833281 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:30:53.366027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:53.363451 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" podUID="4d20bf7a-be4f-48b3-b9b0-d39ca9320150" containerName="payload-processing" containerID="cri-o://6ef430a95fb48e42af16563c46f2cad2fc0e9d7292e3084271854fbbd96912f0" gracePeriod=25 Oct 30 19:30:53.580344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:53.580308 5047 generic.go:356] "Generic (PLEG): container finished" podID="4d20bf7a-be4f-48b3-b9b0-d39ca9320150" containerID="6ef430a95fb48e42af16563c46f2cad2fc0e9d7292e3084271854fbbd96912f0" exitCode=0 Oct 30 19:30:53.580959 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:53.580930 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" event={"ID":"4d20bf7a-be4f-48b3-b9b0-d39ca9320150","Type":"ContainerDied","Data":"6ef430a95fb48e42af16563c46f2cad2fc0e9d7292e3084271854fbbd96912f0"} Oct 30 19:30:53.730872 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:53.730841 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" Oct 30 19:30:53.825630 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:53.825592 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d20bf7a-be4f-48b3-b9b0-d39ca9320150-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d20bf7a-be4f-48b3-b9b0-d39ca9320150-plugins-config-volume\") pod \"4d20bf7a-be4f-48b3-b9b0-d39ca9320150\" (UID: \"4d20bf7a-be4f-48b3-b9b0-d39ca9320150\") " Oct 30 19:30:53.826040 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:53.825982 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d20bf7a-be4f-48b3-b9b0-d39ca9320150-kube-api-access-6qkj4\" (UniqueName: \"kubernetes.io/projected/4d20bf7a-be4f-48b3-b9b0-d39ca9320150-kube-api-access-6qkj4\") pod \"4d20bf7a-be4f-48b3-b9b0-d39ca9320150\" (UID: \"4d20bf7a-be4f-48b3-b9b0-d39ca9320150\") " Oct 30 19:30:53.826127 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:53.826076 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d20bf7a-be4f-48b3-b9b0-d39ca9320150-plugins-config-volume" pod "4d20bf7a-be4f-48b3-b9b0-d39ca9320150" (UID: "4d20bf7a-be4f-48b3-b9b0-d39ca9320150"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:53.826721 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:53.826697 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d20bf7a-be4f-48b3-b9b0-d39ca9320150-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:30:53.828183 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:53.828157 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d20bf7a-be4f-48b3-b9b0-d39ca9320150-kube-api-access-6qkj4" pod "4d20bf7a-be4f-48b3-b9b0-d39ca9320150" (UID: "4d20bf7a-be4f-48b3-b9b0-d39ca9320150"). InnerVolumeSpecName "kube-api-access-6qkj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:53.928164 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:53.928130 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qkj4\" (UniqueName: \"kubernetes.io/projected/4d20bf7a-be4f-48b3-b9b0-d39ca9320150-kube-api-access-6qkj4\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:30:54.591457 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:54.591423 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" Oct 30 19:30:54.591890 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:54.591451 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb" event={"ID":"4d20bf7a-be4f-48b3-b9b0-d39ca9320150","Type":"ContainerDied","Data":"29de9a2273791d4cb5524a5d24c6611cb523305a6fe4b0a7fe0f5e886bc336bd"} Oct 30 19:30:54.591890 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:54.591525 5047 scope.go:122] "RemoveContainer" containerID="6ef430a95fb48e42af16563c46f2cad2fc0e9d7292e3084271854fbbd96912f0" Oct 30 19:30:54.668000 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:54.667957 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb"] Oct 30 19:30:54.676730 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:54.676695 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f215f8b0-749ddf5f56-hthjb"] Oct 30 19:30:55.218622 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:55.218589 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d20bf7a-be4f-48b3-b9b0-d39ca9320150" path="/var/lib/kubelet/pods/4d20bf7a-be4f-48b3-b9b0-d39ca9320150/volumes" Oct 30 19:30:57.972030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:57.941227 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx"] Oct 30 19:30:58.026036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:58.019502 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" podUID="73b43cbc-ec7e-4388-ba6f-9523c56d33b9" containerName="istio-proxy" containerID="cri-o://0dfb2bffac4021b489b749b1666862ca62645ff98b43f372c9136dcff4e8bbb0" gracePeriod=30 Oct 30 19:30:59.629029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:30:59.626534 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 19:31:00.344220 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:00.344175 5047 patch_prober.go:29] interesting pod/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:00.344396 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:00.344246 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" podUID="592dcc49-fe49-4168-aa60-b51052026387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:02.075473 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:02.075439 5047 generic.go:356] "Generic (PLEG): container finished" podID="592dcc49-fe49-4168-aa60-b51052026387" containerID="e6e67279b91b253402119aa734fc4970f15cbbe113740ea65ec347659e66e7af" exitCode=0 Oct 30 19:31:02.075931 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:02.075518 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" event={"ID":"592dcc49-fe49-4168-aa60-b51052026387","Type":"ContainerDied","Data":"e6e67279b91b253402119aa734fc4970f15cbbe113740ea65ec347659e66e7af"} Oct 30 19:31:02.076159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:02.076140 5047 scope.go:122] "RemoveContainer" containerID="e6e67279b91b253402119aa734fc4970f15cbbe113740ea65ec347659e66e7af" Oct 30 19:31:02.078017 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:02.077974 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:31:02.078111 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:02.078063 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:31:02.078150 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:02.078111 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:31:03.087054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:03.086979 5047 generic.go:356] "Generic (PLEG): container finished" podID="73b43cbc-ec7e-4388-ba6f-9523c56d33b9" containerID="0dfb2bffac4021b489b749b1666862ca62645ff98b43f372c9136dcff4e8bbb0" exitCode=0 Oct 30 19:31:03.087499 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:03.087056 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" event={"ID":"73b43cbc-ec7e-4388-ba6f-9523c56d33b9","Type":"ContainerDied","Data":"0dfb2bffac4021b489b749b1666862ca62645ff98b43f372c9136dcff4e8bbb0"} Oct 30 19:31:03.087499 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:03.087115 5047 scope.go:122] "RemoveContainer" containerID="d2491b1a0e4e45bc332c4f5efd18e50f9edad8a5a2731c007707b0dea4801366" Oct 30 19:31:03.089423 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:03.089399 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" event={"ID":"592dcc49-fe49-4168-aa60-b51052026387","Type":"ContainerStarted","Data":"c04677a04c89b21375b7f52e2f7b77d6a55cf52edb3b39fc63c16c43c6319cf8"} Oct 30 19:31:03.817794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:03.817762 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:31:03.817983 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:03.817899 5047 patch_prober.go:29] interesting pod/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:31:03.818179 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:03.817982 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" podUID="592dcc49-fe49-4168-aa60-b51052026387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 19:31:03.961821 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:03.961789 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:31:04.007642 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.007605 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istiod-ca-cert\") pod \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " Oct 30 19:31:04.007861 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.007659 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-credential-socket\") pod \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " Oct 30 19:31:04.007936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.007887 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-envoy\") pod \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " Oct 30 19:31:04.007936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.007913 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-workload-socket\") pod \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " Oct 30 19:31:04.008071 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.007972 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-podinfo\") pod \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " Oct 30 19:31:04.008125 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.008082 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-credential-socket" pod "73b43cbc-ec7e-4388-ba6f-9523c56d33b9" (UID: "73b43cbc-ec7e-4388-ba6f-9523c56d33b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:04.008125 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.008103 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istiod-ca-cert" pod "73b43cbc-ec7e-4388-ba6f-9523c56d33b9" (UID: "73b43cbc-ec7e-4388-ba6f-9523c56d33b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:04.008229 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.008141 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-workload-certs\") pod \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " Oct 30 19:31:04.008283 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.008245 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-workload-socket" pod "73b43cbc-ec7e-4388-ba6f-9523c56d33b9" (UID: "73b43cbc-ec7e-4388-ba6f-9523c56d33b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:04.008362 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.008342 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-token\" (UniqueName: \"kubernetes.io/projected/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-token\") pod \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " Oct 30 19:31:04.008422 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.008359 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-workload-certs" pod "73b43cbc-ec7e-4388-ba6f-9523c56d33b9" (UID: "73b43cbc-ec7e-4388-ba6f-9523c56d33b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:04.008422 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.008402 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-data\") pod \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " Oct 30 19:31:04.008604 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.008424 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-kube-api-access-9xx6z\" (UniqueName: \"kubernetes.io/projected/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-kube-api-access-9xx6z\") pod \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\" (UID: \"73b43cbc-ec7e-4388-ba6f-9523c56d33b9\") " Oct 30 19:31:04.008657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.008603 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-data" pod "73b43cbc-ec7e-4388-ba6f-9523c56d33b9" (UID: "73b43cbc-ec7e-4388-ba6f-9523c56d33b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:04.009381 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.009352 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:04.009509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.009384 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:04.009509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.009398 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:04.009509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.009410 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:04.009509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.009423 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:04.010435 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.010407 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-envoy" pod "73b43cbc-ec7e-4388-ba6f-9523c56d33b9" (UID: "73b43cbc-ec7e-4388-ba6f-9523c56d33b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:04.010621 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.010603 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-podinfo" pod "73b43cbc-ec7e-4388-ba6f-9523c56d33b9" (UID: "73b43cbc-ec7e-4388-ba6f-9523c56d33b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:31:04.010715 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.010666 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-token" pod "73b43cbc-ec7e-4388-ba6f-9523c56d33b9" (UID: "73b43cbc-ec7e-4388-ba6f-9523c56d33b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:04.010715 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.010698 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-kube-api-access-9xx6z" pod "73b43cbc-ec7e-4388-ba6f-9523c56d33b9" (UID: "73b43cbc-ec7e-4388-ba6f-9523c56d33b9"). InnerVolumeSpecName "kube-api-access-9xx6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:04.104064 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.103716 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" Oct 30 19:31:04.104064 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.103748 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx" event={"ID":"73b43cbc-ec7e-4388-ba6f-9523c56d33b9","Type":"ContainerDied","Data":"85bd3cfb180b0ab8b198d254d943e1157e0e9d5ec8206b69c5abedbf97432a73"} Oct 30 19:31:04.104064 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.103816 5047 scope.go:122] "RemoveContainer" containerID="0dfb2bffac4021b489b749b1666862ca62645ff98b43f372c9136dcff4e8bbb0" Oct 30 19:31:04.111225 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.111195 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:04.111225 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.111229 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xx6z\" (UniqueName: \"kubernetes.io/projected/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-kube-api-access-9xx6z\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:04.111440 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.111243 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:04.111440 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.111259 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b43cbc-ec7e-4388-ba6f-9523c56d33b9-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:04.164169 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.164068 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx"] Oct 30 19:31:04.168316 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.168288 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f215f8b0-gw-openshift-default-5b9d776c5c-gl5tx"] Oct 30 19:31:04.815452 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.815401 5047 patch_prober.go:29] interesting pod/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:31:04.815669 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:04.815471 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" podUID="592dcc49-fe49-4168-aa60-b51052026387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 19:31:05.078172 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:05.078072 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b43cbc-ec7e-4388-ba6f-9523c56d33b9" path="/var/lib/kubelet/pods/73b43cbc-ec7e-4388-ba6f-9523c56d33b9/volumes" Oct 30 19:31:05.815635 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:05.815590 5047 patch_prober.go:29] interesting pod/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:31:05.816175 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:05.815663 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" podUID="592dcc49-fe49-4168-aa60-b51052026387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 19:31:06.815985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:06.815947 5047 patch_prober.go:29] interesting pod/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:31:06.816416 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:06.816081 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" podUID="592dcc49-fe49-4168-aa60-b51052026387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 19:31:07.815609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:07.815570 5047 patch_prober.go:29] interesting pod/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:31:07.815816 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:07.815635 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" podUID="592dcc49-fe49-4168-aa60-b51052026387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 19:31:08.819455 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:08.819420 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:31:08.819961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:08.819944 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:31:08.820991 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:08.820967 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:31:09.604820 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:31:09.604783 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:31:20.941069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:20.941024 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb"] Oct 30 19:31:21.177938 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.177896 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph"] Oct 30 19:31:21.179417 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.179379 5047 cpu_manager.go:414] "containerMap: removing container" podUID="73b43cbc-ec7e-4388-ba6f-9523c56d33b9" containerName="istio-proxy" Oct 30 19:31:21.179417 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.179415 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b43cbc-ec7e-4388-ba6f-9523c56d33b9" containerName="istio-proxy" Oct 30 19:31:21.179637 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.179436 5047 cpu_manager.go:414] "containerMap: removing container" podUID="73b43cbc-ec7e-4388-ba6f-9523c56d33b9" containerName="istio-proxy" Oct 30 19:31:21.179637 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.179443 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b43cbc-ec7e-4388-ba6f-9523c56d33b9" containerName="istio-proxy" Oct 30 19:31:21.179637 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.179478 5047 cpu_manager.go:414] "containerMap: removing container" podUID="4d20bf7a-be4f-48b3-b9b0-d39ca9320150" containerName="payload-processing" Oct 30 19:31:21.179637 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.179485 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d20bf7a-be4f-48b3-b9b0-d39ca9320150" containerName="payload-processing" Oct 30 19:31:21.179855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.179674 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d20bf7a-be4f-48b3-b9b0-d39ca9320150" containerName="payload-processing" Oct 30 19:31:21.179855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.179706 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b43cbc-ec7e-4388-ba6f-9523c56d33b9" containerName="istio-proxy" Oct 30 19:31:21.179855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.179715 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b43cbc-ec7e-4388-ba6f-9523c56d33b9" containerName="istio-proxy" Oct 30 19:31:21.185998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.185967 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.199121 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.198959 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph"] Oct 30 19:31:21.201335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.201248 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-f83282-openshift-default-dockercfg-4lzlr\"" Oct 30 19:31:21.265066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.265000 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-envoy\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.265272 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.265084 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istiod-ca-cert\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.265272 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.265105 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-credential-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.265272 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.265135 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-data\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.265432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.265377 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-workload-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.265504 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.265483 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-podinfo\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.265740 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.265714 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-workload-certs\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.265807 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.265791 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg5ck\" (UniqueName: \"kubernetes.io/projected/e5345813-fe79-4f6a-a6dc-489770c4e2f6-kube-api-access-lg5ck\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.265861 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.265818 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-token\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.419477 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.386809 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg5ck\" (UniqueName: \"kubernetes.io/projected/e5345813-fe79-4f6a-a6dc-489770c4e2f6-kube-api-access-lg5ck\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.449197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.449101 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-token\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.493062 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.482190 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-envoy\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.549036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.525583 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istiod-ca-cert\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.550609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.550566 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-credential-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.560356 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.467256 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg5ck\" (UniqueName: \"kubernetes.io/projected/e5345813-fe79-4f6a-a6dc-489770c4e2f6-kube-api-access-lg5ck\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.560882 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.551128 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-credential-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.561263 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.530925 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istiod-ca-cert\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.561438 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.560510 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-data\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.561626 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.560885 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-data\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.561954 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.561934 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-workload-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.562472 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.562369 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-workload-socket\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.562716 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.562699 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-podinfo\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.563405 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.563384 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-workload-certs\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.565102 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.563798 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-workload-certs\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.568326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.568276 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-podinfo\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.644095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.625371 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-envoy\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.660033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.654030 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-token\") pod \"e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:21.829082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:21.819319 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:22.822738 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:31:22.822706 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5345813_fe79_4f6a_a6dc_489770c4e2f6.slice/crio-5950a5a05acaad8066c61b2353059f2e3cb32bcc727389ac4262e729cfe013df WatchSource:0}: Error finding container 5950a5a05acaad8066c61b2353059f2e3cb32bcc727389ac4262e729cfe013df: Status 404 returned error can't find the container with id 5950a5a05acaad8066c61b2353059f2e3cb32bcc727389ac4262e729cfe013df Oct 30 19:31:22.823232 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:22.822818 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph"] Oct 30 19:31:22.825503 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:22.825469 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:31:22.825606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:22.825542 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:31:22.825606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:22.825574 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:31:23.616407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:23.616366 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" event={"ID":"e5345813-fe79-4f6a-a6dc-489770c4e2f6","Type":"ContainerStarted","Data":"b12d72f7f657ff5d176c4fbe6d49412e64d5742ac47634f61704dd58b9b096e6"} Oct 30 19:31:23.616601 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:23.616435 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" event={"ID":"e5345813-fe79-4f6a-a6dc-489770c4e2f6","Type":"ContainerStarted","Data":"5950a5a05acaad8066c61b2353059f2e3cb32bcc727389ac4262e729cfe013df"} Oct 30 19:31:23.670760 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:23.670703 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" podStartSLOduration=2.670687171 podStartE2EDuration="2.670687171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:23.669353252 +0000 UTC m=+1636.743808613" watchObservedRunningTime="2026-10-30 19:31:23.670687171 +0000 UTC m=+1636.745142529" Oct 30 19:31:23.824516 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:23.824412 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:23.870368 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:23.865907 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:24.624016 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:24.623976 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:24.625116 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:24.625094 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:31:25.942132 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:25.942060 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" podUID="c3098017-4e51-4b43-834d-6c2449565f6b" containerName="payload-pre-processing" containerID="cri-o://195613bc953a575153b1556cfbdf243b43c85b9356d234ec64c81b947ddd6f80" gracePeriod=25 Oct 30 19:31:26.240781 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:26.240750 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" Oct 30 19:31:26.393506 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:26.393467 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3098017-4e51-4b43-834d-6c2449565f6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3098017-4e51-4b43-834d-6c2449565f6b-plugins-config-volume\") pod \"c3098017-4e51-4b43-834d-6c2449565f6b\" (UID: \"c3098017-4e51-4b43-834d-6c2449565f6b\") " Oct 30 19:31:26.393689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:26.393601 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3098017-4e51-4b43-834d-6c2449565f6b-kube-api-access-4xdwv\" (UniqueName: \"kubernetes.io/projected/c3098017-4e51-4b43-834d-6c2449565f6b-kube-api-access-4xdwv\") pod \"c3098017-4e51-4b43-834d-6c2449565f6b\" (UID: \"c3098017-4e51-4b43-834d-6c2449565f6b\") " Oct 30 19:31:26.393883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:26.393858 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3098017-4e51-4b43-834d-6c2449565f6b-plugins-config-volume" pod "c3098017-4e51-4b43-834d-6c2449565f6b" (UID: "c3098017-4e51-4b43-834d-6c2449565f6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:26.394431 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:26.394408 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3098017-4e51-4b43-834d-6c2449565f6b-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:26.395772 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:26.395747 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3098017-4e51-4b43-834d-6c2449565f6b-kube-api-access-4xdwv" pod "c3098017-4e51-4b43-834d-6c2449565f6b" (UID: "c3098017-4e51-4b43-834d-6c2449565f6b"). InnerVolumeSpecName "kube-api-access-4xdwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:26.496382 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:26.496287 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xdwv\" (UniqueName: \"kubernetes.io/projected/c3098017-4e51-4b43-834d-6c2449565f6b-kube-api-access-4xdwv\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:26.687232 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:26.687194 5047 generic.go:356] "Generic (PLEG): container finished" podID="c3098017-4e51-4b43-834d-6c2449565f6b" containerID="195613bc953a575153b1556cfbdf243b43c85b9356d234ec64c81b947ddd6f80" exitCode=0 Oct 30 19:31:26.687441 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:26.687244 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" Oct 30 19:31:26.687441 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:26.687276 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" event={"ID":"c3098017-4e51-4b43-834d-6c2449565f6b","Type":"ContainerDied","Data":"195613bc953a575153b1556cfbdf243b43c85b9356d234ec64c81b947ddd6f80"} Oct 30 19:31:26.687441 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:26.687332 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb" event={"ID":"c3098017-4e51-4b43-834d-6c2449565f6b","Type":"ContainerDied","Data":"105c84e666541b4dcde026953c0888bb4626800fa3242fa9206e7ccc0b521316"} Oct 30 19:31:26.687441 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:26.687356 5047 scope.go:122] "RemoveContainer" containerID="195613bc953a575153b1556cfbdf243b43c85b9356d234ec64c81b947ddd6f80" Oct 30 19:31:26.705656 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:26.705628 5047 scope.go:122] "RemoveContainer" containerID="195613bc953a575153b1556cfbdf243b43c85b9356d234ec64c81b947ddd6f80" Oct 30 19:31:26.706073 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:31:26.706046 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195613bc953a575153b1556cfbdf243b43c85b9356d234ec64c81b947ddd6f80\": container with ID starting with 195613bc953a575153b1556cfbdf243b43c85b9356d234ec64c81b947ddd6f80 not found: ID does not exist" containerID="195613bc953a575153b1556cfbdf243b43c85b9356d234ec64c81b947ddd6f80" Oct 30 19:31:26.706191 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:26.706086 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195613bc953a575153b1556cfbdf243b43c85b9356d234ec64c81b947ddd6f80"} err="failed to get container status \"195613bc953a575153b1556cfbdf243b43c85b9356d234ec64c81b947ddd6f80\": rpc error: code = NotFound desc = could not find container \"195613bc953a575153b1556cfbdf243b43c85b9356d234ec64c81b947ddd6f80\": container with ID starting with 195613bc953a575153b1556cfbdf243b43c85b9356d234ec64c81b947ddd6f80 not found: ID does not exist" Oct 30 19:31:26.739861 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:26.739819 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb"] Oct 30 19:31:26.743955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:26.743916 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3bdb4fd2-684b7fdff-bt4qb"] Oct 30 19:31:27.054590 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:27.054547 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3098017-4e51-4b43-834d-6c2449565f6b" path="/var/lib/kubelet/pods/c3098017-4e51-4b43-834d-6c2449565f6b/volumes" Oct 30 19:31:39.784030 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:31:39.777607 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:31:40.800055 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:40.799591 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq"] Oct 30 19:31:40.800653 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:40.800362 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" podUID="592dcc49-fe49-4168-aa60-b51052026387" containerName="istio-proxy" containerID="cri-o://c04677a04c89b21375b7f52e2f7b77d6a55cf52edb3b39fc63c16c43c6319cf8" gracePeriod=30 Oct 30 19:31:42.339081 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.333724 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf"] Oct 30 19:31:42.477076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.460298 5047 cpu_manager.go:414] "containerMap: removing container" podUID="c3098017-4e51-4b43-834d-6c2449565f6b" containerName="payload-pre-processing" Oct 30 19:31:42.477076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.460337 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3098017-4e51-4b43-834d-6c2449565f6b" containerName="payload-pre-processing" Oct 30 19:31:42.477076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.460609 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3098017-4e51-4b43-834d-6c2449565f6b" containerName="payload-pre-processing" Oct 30 19:31:42.539609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.539563 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf"] Oct 30 19:31:42.625979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.624775 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.625979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.625213 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2123dbbb-db97-46f3-b265-a81414f0652c-istio-token\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.625979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.625313 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-istio-envoy\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.625979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.625422 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br7h7\" (UniqueName: \"kubernetes.io/projected/2123dbbb-db97-46f3-b265-a81414f0652c-kube-api-access-br7h7\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.625979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.625467 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2123dbbb-db97-46f3-b265-a81414f0652c-istio-podinfo\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.625979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.625539 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-istio-data\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.625979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.625633 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2123dbbb-db97-46f3-b265-a81414f0652c-istiod-ca-cert\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.625979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.625679 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-credential-socket\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.625979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.625801 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-workload-socket\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.625979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.625862 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-workload-certs\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.702930 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.658068 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-6b7ccc-openshift-default-dockercfg-8b86q\"" Oct 30 19:31:42.778033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.777594 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-workload-socket\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.778033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.777690 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-workload-certs\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.778033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.777736 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2123dbbb-db97-46f3-b265-a81414f0652c-istio-token\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.778033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.777813 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-istio-envoy\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.778033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.777970 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br7h7\" (UniqueName: \"kubernetes.io/projected/2123dbbb-db97-46f3-b265-a81414f0652c-kube-api-access-br7h7\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.778033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.778033 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2123dbbb-db97-46f3-b265-a81414f0652c-istio-podinfo\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.778736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.778113 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-istio-data\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.778736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.778208 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2123dbbb-db97-46f3-b265-a81414f0652c-istiod-ca-cert\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.778736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.778259 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-credential-socket\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.778945 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.778909 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-credential-socket\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.802031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.779282 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-workload-socket\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.802031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.779541 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-workload-certs\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.802031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.797500 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-istio-data\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.802031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.798766 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2123dbbb-db97-46f3-b265-a81414f0652c-istiod-ca-cert\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.836030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.811773 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2123dbbb-db97-46f3-b265-a81414f0652c-istio-token\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.853959 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.853921 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br7h7\" (UniqueName: \"kubernetes.io/projected/2123dbbb-db97-46f3-b265-a81414f0652c-kube-api-access-br7h7\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.904084 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.899758 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2123dbbb-db97-46f3-b265-a81414f0652c-istio-podinfo\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.957680 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.957630 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-istio-envoy\") pod \"e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:42.996486 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:42.996441 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:43.070074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:43.070035 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:43.638059 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:43.596865 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:43.678242 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:43.665692 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:31:43.943082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:43.936203 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:43.987033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:43.961038 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:44.172573 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:44.126513 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:44.173432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:44.156121 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:44.173934 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:44.173908 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:31:44.512031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:44.508691 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf"] Oct 30 19:31:44.737685 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:44.733620 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:31:44.737685 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:44.733732 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:31:44.737685 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:44.733775 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:31:44.797486 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:44.797112 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:31:45.310051 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:45.309909 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" event={"ID":"2123dbbb-db97-46f3-b265-a81414f0652c","Type":"ContainerStarted","Data":"36890bb1da7aa5b57502bfc2c25147c106279f5ffd1cda27275fd7443e0f8614"} Oct 30 19:31:45.310713 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:45.310352 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" event={"ID":"2123dbbb-db97-46f3-b265-a81414f0652c","Type":"ContainerStarted","Data":"dd71ae2669327f8a830b87b36e71f7bca7a88d4323250398f619c0e58e3363ec"} Oct 30 19:31:45.363199 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:45.363130 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" podStartSLOduration=3.363107359 podStartE2EDuration="3.363107359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:45.360080012 +0000 UTC m=+1658.434535373" watchObservedRunningTime="2026-10-30 19:31:45.363107359 +0000 UTC m=+1658.437562719" Oct 30 19:31:46.008446 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.008403 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:46.016436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.016403 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:46.171363 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.170925 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:31:46.329082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.327206 5047 generic.go:356] "Generic (PLEG): container finished" podID="592dcc49-fe49-4168-aa60-b51052026387" containerID="c04677a04c89b21375b7f52e2f7b77d6a55cf52edb3b39fc63c16c43c6319cf8" exitCode=0 Oct 30 19:31:46.329082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.327256 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" Oct 30 19:31:46.329082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.327300 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" event={"ID":"592dcc49-fe49-4168-aa60-b51052026387","Type":"ContainerDied","Data":"c04677a04c89b21375b7f52e2f7b77d6a55cf52edb3b39fc63c16c43c6319cf8"} Oct 30 19:31:46.329082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.327350 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq" event={"ID":"592dcc49-fe49-4168-aa60-b51052026387","Type":"ContainerDied","Data":"558c90378e010fa3f92ed0d0b113324aab2443927392a4d83fdfbdd025972113"} Oct 30 19:31:46.329082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.327403 5047 scope.go:122] "RemoveContainer" containerID="c04677a04c89b21375b7f52e2f7b77d6a55cf52edb3b39fc63c16c43c6319cf8" Oct 30 19:31:46.329082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.327755 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:46.329478 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.329176 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:31:46.338982 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.338956 5047 scope.go:122] "RemoveContainer" containerID="e6e67279b91b253402119aa734fc4970f15cbbe113740ea65ec347659e66e7af" Oct 30 19:31:46.354990 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.354960 5047 scope.go:122] "RemoveContainer" containerID="c04677a04c89b21375b7f52e2f7b77d6a55cf52edb3b39fc63c16c43c6319cf8" Oct 30 19:31:46.355407 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:31:46.355381 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c04677a04c89b21375b7f52e2f7b77d6a55cf52edb3b39fc63c16c43c6319cf8\": container with ID starting with c04677a04c89b21375b7f52e2f7b77d6a55cf52edb3b39fc63c16c43c6319cf8 not found: ID does not exist" containerID="c04677a04c89b21375b7f52e2f7b77d6a55cf52edb3b39fc63c16c43c6319cf8" Oct 30 19:31:46.355516 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.355419 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c04677a04c89b21375b7f52e2f7b77d6a55cf52edb3b39fc63c16c43c6319cf8"} err="failed to get container status \"c04677a04c89b21375b7f52e2f7b77d6a55cf52edb3b39fc63c16c43c6319cf8\": rpc error: code = NotFound desc = could not find container \"c04677a04c89b21375b7f52e2f7b77d6a55cf52edb3b39fc63c16c43c6319cf8\": container with ID starting with c04677a04c89b21375b7f52e2f7b77d6a55cf52edb3b39fc63c16c43c6319cf8 not found: ID does not exist" Oct 30 19:31:46.355516 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.355443 5047 scope.go:122] "RemoveContainer" containerID="e6e67279b91b253402119aa734fc4970f15cbbe113740ea65ec347659e66e7af" Oct 30 19:31:46.355780 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:31:46.355754 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e67279b91b253402119aa734fc4970f15cbbe113740ea65ec347659e66e7af\": container with ID starting with e6e67279b91b253402119aa734fc4970f15cbbe113740ea65ec347659e66e7af not found: ID does not exist" containerID="e6e67279b91b253402119aa734fc4970f15cbbe113740ea65ec347659e66e7af" Oct 30 19:31:46.355849 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.355791 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e67279b91b253402119aa734fc4970f15cbbe113740ea65ec347659e66e7af"} err="failed to get container status \"e6e67279b91b253402119aa734fc4970f15cbbe113740ea65ec347659e66e7af\": rpc error: code = NotFound desc = could not find container \"e6e67279b91b253402119aa734fc4970f15cbbe113740ea65ec347659e66e7af\": container with ID starting with e6e67279b91b253402119aa734fc4970f15cbbe113740ea65ec347659e66e7af not found: ID does not exist" Oct 30 19:31:46.356865 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.356835 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-workload-socket\") pod \"592dcc49-fe49-4168-aa60-b51052026387\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " Oct 30 19:31:46.356993 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.356939 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592dcc49-fe49-4168-aa60-b51052026387-istio-token\" (UniqueName: \"kubernetes.io/projected/592dcc49-fe49-4168-aa60-b51052026387-istio-token\") pod \"592dcc49-fe49-4168-aa60-b51052026387\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " Oct 30 19:31:46.357228 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.357206 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-workload-socket" pod "592dcc49-fe49-4168-aa60-b51052026387" (UID: "592dcc49-fe49-4168-aa60-b51052026387"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:46.357474 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.357456 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/592dcc49-fe49-4168-aa60-b51052026387-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592dcc49-fe49-4168-aa60-b51052026387-istiod-ca-cert\") pod \"592dcc49-fe49-4168-aa60-b51052026387\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " Oct 30 19:31:46.357997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.357541 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-credential-socket\") pod \"592dcc49-fe49-4168-aa60-b51052026387\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " Oct 30 19:31:46.357997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.357580 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592dcc49-fe49-4168-aa60-b51052026387-kube-api-access-4t6h5\" (UniqueName: \"kubernetes.io/projected/592dcc49-fe49-4168-aa60-b51052026387-kube-api-access-4t6h5\") pod \"592dcc49-fe49-4168-aa60-b51052026387\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " Oct 30 19:31:46.357997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.357655 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-istio-data\") pod \"592dcc49-fe49-4168-aa60-b51052026387\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " Oct 30 19:31:46.357997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.357674 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/592dcc49-fe49-4168-aa60-b51052026387-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592dcc49-fe49-4168-aa60-b51052026387-istio-podinfo\") pod \"592dcc49-fe49-4168-aa60-b51052026387\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " Oct 30 19:31:46.357997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.357704 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-istio-envoy\") pod \"592dcc49-fe49-4168-aa60-b51052026387\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " Oct 30 19:31:46.357997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.357802 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-workload-certs\") pod \"592dcc49-fe49-4168-aa60-b51052026387\" (UID: \"592dcc49-fe49-4168-aa60-b51052026387\") " Oct 30 19:31:46.357997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.357969 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-istio-data" pod "592dcc49-fe49-4168-aa60-b51052026387" (UID: "592dcc49-fe49-4168-aa60-b51052026387"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:46.358447 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.358319 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-workload-certs" pod "592dcc49-fe49-4168-aa60-b51052026387" (UID: "592dcc49-fe49-4168-aa60-b51052026387"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:46.358447 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.358341 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-credential-socket" pod "592dcc49-fe49-4168-aa60-b51052026387" (UID: "592dcc49-fe49-4168-aa60-b51052026387"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:46.358447 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.358380 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592dcc49-fe49-4168-aa60-b51052026387-istiod-ca-cert" pod "592dcc49-fe49-4168-aa60-b51052026387" (UID: "592dcc49-fe49-4168-aa60-b51052026387"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:46.359377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.359092 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:46.359377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.359119 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:46.359377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.359133 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592dcc49-fe49-4168-aa60-b51052026387-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:46.359377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.359145 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:46.359377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.359158 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:46.360672 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.360648 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592dcc49-fe49-4168-aa60-b51052026387-istio-token" pod "592dcc49-fe49-4168-aa60-b51052026387" (UID: "592dcc49-fe49-4168-aa60-b51052026387"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:46.360746 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.360682 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592dcc49-fe49-4168-aa60-b51052026387-kube-api-access-4t6h5" pod "592dcc49-fe49-4168-aa60-b51052026387" (UID: "592dcc49-fe49-4168-aa60-b51052026387"). InnerVolumeSpecName "kube-api-access-4t6h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:46.360746 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.360707 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-istio-envoy" pod "592dcc49-fe49-4168-aa60-b51052026387" (UID: "592dcc49-fe49-4168-aa60-b51052026387"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:46.361371 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.361349 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/592dcc49-fe49-4168-aa60-b51052026387-istio-podinfo" pod "592dcc49-fe49-4168-aa60-b51052026387" (UID: "592dcc49-fe49-4168-aa60-b51052026387"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:31:46.460323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.460224 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592dcc49-fe49-4168-aa60-b51052026387-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:46.460323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.460266 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592dcc49-fe49-4168-aa60-b51052026387-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:46.460323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.460279 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/592dcc49-fe49-4168-aa60-b51052026387-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:46.460323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.460292 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t6h5\" (UniqueName: \"kubernetes.io/projected/592dcc49-fe49-4168-aa60-b51052026387-kube-api-access-4t6h5\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:31:46.677769 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.677728 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq"] Oct 30 19:31:46.688947 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:46.688904 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3bdb4fd2-openshift-default-5cd85b8dd9-z2xhq"] Oct 30 19:31:47.053450 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:47.053409 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="592dcc49-fe49-4168-aa60-b51052026387" path="/var/lib/kubelet/pods/592dcc49-fe49-4168-aa60-b51052026387/volumes" Oct 30 19:31:49.910757 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:49.910676 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45"] Oct 30 19:31:49.911454 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:49.911435 5047 cpu_manager.go:414] "containerMap: removing container" podUID="592dcc49-fe49-4168-aa60-b51052026387" containerName="istio-proxy" Oct 30 19:31:49.911454 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:49.911454 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="592dcc49-fe49-4168-aa60-b51052026387" containerName="istio-proxy" Oct 30 19:31:49.911536 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:49.911487 5047 cpu_manager.go:414] "containerMap: removing container" podUID="592dcc49-fe49-4168-aa60-b51052026387" containerName="istio-proxy" Oct 30 19:31:49.911536 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:49.911494 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="592dcc49-fe49-4168-aa60-b51052026387" containerName="istio-proxy" Oct 30 19:31:49.911627 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:49.911618 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="592dcc49-fe49-4168-aa60-b51052026387" containerName="istio-proxy" Oct 30 19:31:49.916814 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:49.916790 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" Oct 30 19:31:49.948052 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:49.947997 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45"] Oct 30 19:31:49.951317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:49.951292 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-6b7ccc\"" Oct 30 19:31:49.993232 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:49.993181 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9zwx\" (UniqueName: \"kubernetes.io/projected/d1e68671-670d-4915-ba4d-47c184994e6a-kube-api-access-j9zwx\") pod \"payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45\" (UID: \"d1e68671-670d-4915-ba4d-47c184994e6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" Oct 30 19:31:49.993598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:49.993573 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e68671-670d-4915-ba4d-47c184994e6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45\" (UID: \"d1e68671-670d-4915-ba4d-47c184994e6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" Oct 30 19:31:50.021262 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:50.021196 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk"] Oct 30 19:31:50.022216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:50.022198 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="592dcc49-fe49-4168-aa60-b51052026387" containerName="istio-proxy" Oct 30 19:31:50.026141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:50.026117 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" Oct 30 19:31:50.065280 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:50.054183 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-6b7ccc-dockercfg-v5rm4\"" Oct 30 19:31:50.099237 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:50.099138 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdpx8\" (UniqueName: \"kubernetes.io/projected/87b98550-4b44-4b24-8c55-03a8e05f08df-kube-api-access-fdpx8\") pod \"payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk\" (UID: \"87b98550-4b44-4b24-8c55-03a8e05f08df\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" Oct 30 19:31:50.099440 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:50.099376 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b98550-4b44-4b24-8c55-03a8e05f08df-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk\" (UID: \"87b98550-4b44-4b24-8c55-03a8e05f08df\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" Oct 30 19:31:50.099940 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:50.099599 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e68671-670d-4915-ba4d-47c184994e6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45\" (UID: \"d1e68671-670d-4915-ba4d-47c184994e6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" Oct 30 19:31:50.245090 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:50.221511 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk"] Oct 30 19:31:50.314028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:50.313213 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9zwx\" (UniqueName: \"kubernetes.io/projected/d1e68671-670d-4915-ba4d-47c184994e6a-kube-api-access-j9zwx\") pod \"payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45\" (UID: \"d1e68671-670d-4915-ba4d-47c184994e6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" Oct 30 19:31:50.431026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:50.423835 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdpx8\" (UniqueName: \"kubernetes.io/projected/87b98550-4b44-4b24-8c55-03a8e05f08df-kube-api-access-fdpx8\") pod \"payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk\" (UID: \"87b98550-4b44-4b24-8c55-03a8e05f08df\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" Oct 30 19:31:50.431026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:50.423957 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b98550-4b44-4b24-8c55-03a8e05f08df-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk\" (UID: \"87b98550-4b44-4b24-8c55-03a8e05f08df\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" Oct 30 19:31:50.431282 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:50.431089 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b98550-4b44-4b24-8c55-03a8e05f08df-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk\" (UID: \"87b98550-4b44-4b24-8c55-03a8e05f08df\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" Oct 30 19:31:50.451533 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:50.444567 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e68671-670d-4915-ba4d-47c184994e6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45\" (UID: \"d1e68671-670d-4915-ba4d-47c184994e6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" Oct 30 19:31:50.475738 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:50.475701 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdpx8\" (UniqueName: \"kubernetes.io/projected/87b98550-4b44-4b24-8c55-03a8e05f08df-kube-api-access-fdpx8\") pod \"payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk\" (UID: \"87b98550-4b44-4b24-8c55-03a8e05f08df\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" Oct 30 19:31:50.478903 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:50.478868 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9zwx\" (UniqueName: \"kubernetes.io/projected/d1e68671-670d-4915-ba4d-47c184994e6a-kube-api-access-j9zwx\") pod \"payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45\" (UID: \"d1e68671-670d-4915-ba4d-47c184994e6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" Oct 30 19:31:50.531415 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:50.531306 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" Oct 30 19:31:50.765534 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:50.765494 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" Oct 30 19:31:51.009908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:51.009166 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45"] Oct 30 19:31:51.278699 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:31:51.278587 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b98550_4b44_4b24_8c55_03a8e05f08df.slice/crio-b2f7818da9d491216dd195abf21a2c8074446aee7bf48c76d53044b89f70c215 WatchSource:0}: Error finding container b2f7818da9d491216dd195abf21a2c8074446aee7bf48c76d53044b89f70c215: Status 404 returned error can't find the container with id b2f7818da9d491216dd195abf21a2c8074446aee7bf48c76d53044b89f70c215 Oct 30 19:31:51.279615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:51.279585 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk"] Oct 30 19:31:51.476317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:51.476277 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" event={"ID":"87b98550-4b44-4b24-8c55-03a8e05f08df","Type":"ContainerStarted","Data":"86c62f6d2eca6d96ee48d9d091102e0b2eb6fe3e08c3ac42120a954967183d47"} Oct 30 19:31:51.476521 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:51.476360 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" event={"ID":"87b98550-4b44-4b24-8c55-03a8e05f08df","Type":"ContainerStarted","Data":"b2f7818da9d491216dd195abf21a2c8074446aee7bf48c76d53044b89f70c215"} Oct 30 19:31:51.476632 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:51.476602 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" Oct 30 19:31:51.478297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:51.478264 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" event={"ID":"d1e68671-670d-4915-ba4d-47c184994e6a","Type":"ContainerStarted","Data":"96c13429ba605e3212ef6e9f50cd199a7d08a999b13b2b7102abe809e2d0683d"} Oct 30 19:31:51.478446 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:51.478315 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" event={"ID":"d1e68671-670d-4915-ba4d-47c184994e6a","Type":"ContainerStarted","Data":"434bc3c8f77b46f455ace59978f9fbf80919f721060c891f22169ebaa0015637"} Oct 30 19:31:51.478446 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:51.478355 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" Oct 30 19:31:51.516000 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:51.515944 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" podStartSLOduration=2.5159286400000003 podStartE2EDuration="2.51592864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:51.512458143 +0000 UTC m=+1664.586913503" watchObservedRunningTime="2026-10-30 19:31:51.51592864 +0000 UTC m=+1664.590383998" Oct 30 19:31:51.551224 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:31:51.551112 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" podStartSLOduration=2.551092663 podStartE2EDuration="2.551092663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:51.548995799 +0000 UTC m=+1664.623451157" watchObservedRunningTime="2026-10-30 19:31:51.551092663 +0000 UTC m=+1664.625548023" Oct 30 19:32:02.490467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:02.490432 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" Oct 30 19:32:02.490999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:02.490864 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" Oct 30 19:32:09.610048 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:32:09.609998 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:32:10.255851 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:10.249597 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": read tcp 10.132.0.2:38870->10.132.0.104:15021: read: connection reset by peer" start-of-body= Oct 30 19:32:10.255851 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:10.249663 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" podUID="e5345813-fe79-4f6a-a6dc-489770c4e2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": read tcp 10.132.0.2:38870->10.132.0.104:15021: read: connection reset by peer" Oct 30 19:32:10.755803 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:10.755766 5047 generic.go:356] "Generic (PLEG): container finished" podID="e5345813-fe79-4f6a-a6dc-489770c4e2f6" containerID="b12d72f7f657ff5d176c4fbe6d49412e64d5742ac47634f61704dd58b9b096e6" exitCode=0 Oct 30 19:32:10.756264 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:10.755850 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" event={"ID":"e5345813-fe79-4f6a-a6dc-489770c4e2f6","Type":"ContainerDied","Data":"b12d72f7f657ff5d176c4fbe6d49412e64d5742ac47634f61704dd58b9b096e6"} Oct 30 19:32:10.756458 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:10.756438 5047 scope.go:122] "RemoveContainer" containerID="b12d72f7f657ff5d176c4fbe6d49412e64d5742ac47634f61704dd58b9b096e6" Oct 30 19:32:10.758166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:10.758138 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:32:10.758219 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:10.758208 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:32:10.758254 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:10.758238 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:32:11.768515 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:11.768457 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" event={"ID":"e5345813-fe79-4f6a-a6dc-489770c4e2f6","Type":"ContainerStarted","Data":"c2c74b902fd8f0d709b166bc2316bc645f5aa914939ce3254da7761dd689e971"} Oct 30 19:32:11.820299 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:11.820255 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:32:11.820490 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:11.820457 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:32:11.820591 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:11.820526 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" podUID="e5345813-fe79-4f6a-a6dc-489770c4e2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:32:12.820933 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:12.820896 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:32:12.821469 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:12.820966 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" podUID="e5345813-fe79-4f6a-a6dc-489770c4e2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:32:13.686648 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:13.686610 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww"] Oct 30 19:32:13.690356 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:13.690334 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" Oct 30 19:32:13.710877 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:13.710837 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ccf8bd61-dockercfg-96fv9\"" Oct 30 19:32:13.712262 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:13.712234 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww"] Oct 30 19:32:13.718685 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:13.718655 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ccf8bd61\"" Oct 30 19:32:13.838027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:13.823388 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8xdd\" (UniqueName: \"kubernetes.io/projected/820cab28-9ecc-46d5-945a-df0b07cf3c4f-kube-api-access-v8xdd\") pod \"payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww\" (UID: \"820cab28-9ecc-46d5-945a-df0b07cf3c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" Oct 30 19:32:13.838027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:13.823553 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820cab28-9ecc-46d5-945a-df0b07cf3c4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww\" (UID: \"820cab28-9ecc-46d5-945a-df0b07cf3c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" Oct 30 19:32:13.838027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:13.823961 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:32:13.838027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:13.824035 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" podUID="e5345813-fe79-4f6a-a6dc-489770c4e2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:32:13.955088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:13.937732 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8xdd\" (UniqueName: \"kubernetes.io/projected/820cab28-9ecc-46d5-945a-df0b07cf3c4f-kube-api-access-v8xdd\") pod \"payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww\" (UID: \"820cab28-9ecc-46d5-945a-df0b07cf3c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" Oct 30 19:32:13.955088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:13.937907 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820cab28-9ecc-46d5-945a-df0b07cf3c4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww\" (UID: \"820cab28-9ecc-46d5-945a-df0b07cf3c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" Oct 30 19:32:13.955088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:13.939169 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820cab28-9ecc-46d5-945a-df0b07cf3c4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww\" (UID: \"820cab28-9ecc-46d5-945a-df0b07cf3c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" Oct 30 19:32:14.047818 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:14.046738 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8xdd\" (UniqueName: \"kubernetes.io/projected/820cab28-9ecc-46d5-945a-df0b07cf3c4f-kube-api-access-v8xdd\") pod \"payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww\" (UID: \"820cab28-9ecc-46d5-945a-df0b07cf3c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" Oct 30 19:32:14.047818 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:14.047088 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" Oct 30 19:32:14.667974 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:14.667942 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww"] Oct 30 19:32:14.669442 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:32:14.669414 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820cab28_9ecc_46d5_945a_df0b07cf3c4f.slice/crio-733e7a12d01adb799e0b3cf8ffa88a482470f9a13a42db692baed5d2085beea0 WatchSource:0}: Error finding container 733e7a12d01adb799e0b3cf8ffa88a482470f9a13a42db692baed5d2085beea0: Status 404 returned error can't find the container with id 733e7a12d01adb799e0b3cf8ffa88a482470f9a13a42db692baed5d2085beea0 Oct 30 19:32:14.796782 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:14.796748 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" event={"ID":"820cab28-9ecc-46d5-945a-df0b07cf3c4f","Type":"ContainerStarted","Data":"d6477213fc05d20752053e97a964079703e71469b543977a830a37e10d04808d"} Oct 30 19:32:14.796936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:14.796840 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" event={"ID":"820cab28-9ecc-46d5-945a-df0b07cf3c4f","Type":"ContainerStarted","Data":"733e7a12d01adb799e0b3cf8ffa88a482470f9a13a42db692baed5d2085beea0"} Oct 30 19:32:14.797035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:14.796987 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" Oct 30 19:32:14.820914 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:14.820879 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:32:14.821120 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:14.820931 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" podUID="e5345813-fe79-4f6a-a6dc-489770c4e2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:32:14.860882 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:14.858715 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" podStartSLOduration=1.858693551 podStartE2EDuration="1.858693551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:14.856385814 +0000 UTC m=+1687.930841174" watchObservedRunningTime="2026-10-30 19:32:14.858693551 +0000 UTC m=+1687.933148911" Oct 30 19:32:15.820382 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:15.820348 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:32:15.820605 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:15.820419 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" podUID="e5345813-fe79-4f6a-a6dc-489770c4e2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:32:16.820429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:16.820391 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:32:16.820922 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:16.820452 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" podUID="e5345813-fe79-4f6a-a6dc-489770c4e2f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:32:17.843106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:17.837971 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:32:17.843106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:17.839583 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:32:17.843106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:17.840462 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 19:32:25.808909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:25.808871 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" Oct 30 19:32:29.283914 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.283870 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns"] Oct 30 19:32:29.303699 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.303667 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.322338 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.322290 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-891d6b-openshift-default-dockercfg-58nqj\"" Oct 30 19:32:29.377058 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.376965 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns"] Oct 30 19:32:29.543078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.542473 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df4b5\" (UniqueName: \"kubernetes.io/projected/faec438d-9b43-450a-b39e-e6eabb6e0b98-kube-api-access-df4b5\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.543078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.542593 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-workload-socket\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.543078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.542645 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faec438d-9b43-450a-b39e-e6eabb6e0b98-istiod-ca-cert\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.543078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.542711 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-workload-certs\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.543078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.542744 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-credential-socket\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.543078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.542767 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-data\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.543078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.542795 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-podinfo\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.543573 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.543120 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-token\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.543573 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.543161 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-envoy\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.648026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.647371 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-token\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.648026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.647440 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-envoy\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.648026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.647559 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df4b5\" (UniqueName: \"kubernetes.io/projected/faec438d-9b43-450a-b39e-e6eabb6e0b98-kube-api-access-df4b5\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.648026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.647656 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-workload-socket\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.648026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.647702 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faec438d-9b43-450a-b39e-e6eabb6e0b98-istiod-ca-cert\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.648026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.647759 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-workload-certs\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.648026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.647791 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-credential-socket\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.648026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.647816 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-data\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.648026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.647842 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-podinfo\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.667400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.650352 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-data\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.667400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.651045 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faec438d-9b43-450a-b39e-e6eabb6e0b98-istiod-ca-cert\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.667400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.654220 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-credential-socket\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.667400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.654542 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-workload-certs\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.667400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.654751 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-workload-socket\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.667400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.662613 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-podinfo\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.667400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.662929 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-envoy\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.717030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.708889 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df4b5\" (UniqueName: \"kubernetes.io/projected/faec438d-9b43-450a-b39e-e6eabb6e0b98-kube-api-access-df4b5\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.720165 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.720128 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-token\") pod \"e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:29.940400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:29.940361 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:30.397786 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:32:30.397753 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaec438d_9b43_450a_b39e_e6eabb6e0b98.slice/crio-117a75b4281c5925f7a2c876800a2b0bef635eb83b196f03989d635627261caa WatchSource:0}: Error finding container 117a75b4281c5925f7a2c876800a2b0bef635eb83b196f03989d635627261caa: Status 404 returned error can't find the container with id 117a75b4281c5925f7a2c876800a2b0bef635eb83b196f03989d635627261caa Oct 30 19:32:30.398162 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:30.398030 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns"] Oct 30 19:32:30.400532 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:30.400504 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:32:30.400611 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:30.400572 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:32:30.400649 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:30.400620 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:32:31.003391 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:31.003352 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" event={"ID":"faec438d-9b43-450a-b39e-e6eabb6e0b98","Type":"ContainerStarted","Data":"d649a5636675450846e4a3effd0c5f3d2c9efd4b5b1d444076e3d99ee5b21059"} Oct 30 19:32:31.003582 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:31.003446 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" event={"ID":"faec438d-9b43-450a-b39e-e6eabb6e0b98","Type":"ContainerStarted","Data":"117a75b4281c5925f7a2c876800a2b0bef635eb83b196f03989d635627261caa"} Oct 30 19:32:31.089653 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:31.079358 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" podStartSLOduration=2.079332646 podStartE2EDuration="2.079332646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:31.056481438 +0000 UTC m=+1704.130947174" watchObservedRunningTime="2026-10-30 19:32:31.079332646 +0000 UTC m=+1704.153787986" Oct 30 19:32:31.952032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:31.947184 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:31.964330 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:31.964301 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:32.010349 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:32.010317 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:32.011337 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:32.011318 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:32:34.603213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:34.603176 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5"] Oct 30 19:32:34.625727 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:34.625698 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5"] Oct 30 19:32:34.625952 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:34.625816 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:32:34.642095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:34.641841 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ccf8bd61-serving-cert\"" Oct 30 19:32:34.706587 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:34.702264 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lvrg\" (UniqueName: \"kubernetes.io/projected/ccac5abe-13ee-4832-9b5a-38029145f82c-kube-api-access-8lvrg\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5\" (UID: \"ccac5abe-13ee-4832-9b5a-38029145f82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:32:34.706587 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:34.702422 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccac5abe-13ee-4832-9b5a-38029145f82c-maas-api-tls\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5\" (UID: \"ccac5abe-13ee-4832-9b5a-38029145f82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:32:34.706587 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:34.702473 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccac5abe-13ee-4832-9b5a-38029145f82c-metrics-certs\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5\" (UID: \"ccac5abe-13ee-4832-9b5a-38029145f82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:32:34.706587 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:34.702502 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccac5abe-13ee-4832-9b5a-38029145f82c-openshift-service-ca\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5\" (UID: \"ccac5abe-13ee-4832-9b5a-38029145f82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:32:34.842141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:34.842099 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lvrg\" (UniqueName: \"kubernetes.io/projected/ccac5abe-13ee-4832-9b5a-38029145f82c-kube-api-access-8lvrg\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5\" (UID: \"ccac5abe-13ee-4832-9b5a-38029145f82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:32:34.842898 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:34.842858 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccac5abe-13ee-4832-9b5a-38029145f82c-maas-api-tls\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5\" (UID: \"ccac5abe-13ee-4832-9b5a-38029145f82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:32:34.843263 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:34.843245 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccac5abe-13ee-4832-9b5a-38029145f82c-metrics-certs\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5\" (UID: \"ccac5abe-13ee-4832-9b5a-38029145f82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:32:34.853661 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:32:34.843169 5047 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-serving-cert: secret "maas-api-e2e-mt-ccf8bd61-serving-cert" not found Oct 30 19:32:34.853661 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:32:34.853299 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccac5abe-13ee-4832-9b5a-38029145f82c-maas-api-tls podName:ccac5abe-13ee-4832-9b5a-38029145f82c nodeName:}" failed. No retries permitted until 2026-10-30 19:32:35.353221563 +0000 UTC m=+1708.427676903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ccac5abe-13ee-4832-9b5a-38029145f82c-maas-api-tls") pod "maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" (UID: "ccac5abe-13ee-4832-9b5a-38029145f82c") : secret "maas-api-e2e-mt-ccf8bd61-serving-cert" not found Oct 30 19:32:34.855687 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:34.854878 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccac5abe-13ee-4832-9b5a-38029145f82c-openshift-service-ca\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5\" (UID: \"ccac5abe-13ee-4832-9b5a-38029145f82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:32:34.862030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:34.843896 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccac5abe-13ee-4832-9b5a-38029145f82c-openshift-service-ca\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5\" (UID: \"ccac5abe-13ee-4832-9b5a-38029145f82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:32:34.874092 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:34.869725 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccac5abe-13ee-4832-9b5a-38029145f82c-metrics-certs\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5\" (UID: \"ccac5abe-13ee-4832-9b5a-38029145f82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:32:34.883815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:34.883780 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lvrg\" (UniqueName: \"kubernetes.io/projected/ccac5abe-13ee-4832-9b5a-38029145f82c-kube-api-access-8lvrg\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5\" (UID: \"ccac5abe-13ee-4832-9b5a-38029145f82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:32:35.395442 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:35.395399 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccac5abe-13ee-4832-9b5a-38029145f82c-maas-api-tls\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5\" (UID: \"ccac5abe-13ee-4832-9b5a-38029145f82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:32:35.398098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:35.398067 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccac5abe-13ee-4832-9b5a-38029145f82c-maas-api-tls\") pod \"maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5\" (UID: \"ccac5abe-13ee-4832-9b5a-38029145f82c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:32:35.548280 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:35.548247 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:32:35.865916 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:35.865881 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5"] Oct 30 19:32:35.870828 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:32:35.870788 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccac5abe_13ee_4832_9b5a_38029145f82c.slice/crio-81fde51a6aa145c2fa98312574d4ae1d460871ecfaa80cafc304c40e2752e68d WatchSource:0}: Error finding container 81fde51a6aa145c2fa98312574d4ae1d460871ecfaa80cafc304c40e2752e68d: Status 404 returned error can't find the container with id 81fde51a6aa145c2fa98312574d4ae1d460871ecfaa80cafc304c40e2752e68d Oct 30 19:32:36.052157 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:36.052119 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" event={"ID":"ccac5abe-13ee-4832-9b5a-38029145f82c","Type":"ContainerStarted","Data":"81fde51a6aa145c2fa98312574d4ae1d460871ecfaa80cafc304c40e2752e68d"} Oct 30 19:32:37.102078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:37.091476 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" event={"ID":"ccac5abe-13ee-4832-9b5a-38029145f82c","Type":"ContainerStarted","Data":"21049643a74e9fdc6f92dcc606343b4421f1c4bb721d9d28505d81a97ca00235"} Oct 30 19:32:37.102078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:37.091817 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:32:37.135374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:37.135319 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" podStartSLOduration=2.6354938949999998 podStartE2EDuration="3.135303227s" totalImagesPullingTime="499.809332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:37.130702018 +0000 UTC m=+1710.205157378" watchObservedRunningTime="2026-10-30 19:32:37.135303227 +0000 UTC m=+1710.209758586" Oct 30 19:32:38.100123 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:38.100088 5047 generic.go:356] "Generic (PLEG): container finished" podID="2123dbbb-db97-46f3-b265-a81414f0652c" containerID="36890bb1da7aa5b57502bfc2c25147c106279f5ffd1cda27275fd7443e0f8614" exitCode=0 Oct 30 19:32:38.100354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:38.100164 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" event={"ID":"2123dbbb-db97-46f3-b265-a81414f0652c","Type":"ContainerDied","Data":"36890bb1da7aa5b57502bfc2c25147c106279f5ffd1cda27275fd7443e0f8614"} Oct 30 19:32:38.100672 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:38.100656 5047 scope.go:122] "RemoveContainer" containerID="36890bb1da7aa5b57502bfc2c25147c106279f5ffd1cda27275fd7443e0f8614" Oct 30 19:32:38.102317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:38.102292 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:32:38.102556 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:38.102357 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:32:38.102556 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:38.102388 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:32:39.224272 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:39.224214 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" event={"ID":"2123dbbb-db97-46f3-b265-a81414f0652c","Type":"ContainerStarted","Data":"a317c07c3b6d43c6dd7b7f5eb3306529cf080af30733095e8f40c55326b35b28"} Oct 30 19:32:39.929103 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:32:39.921713 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:32:40.075820 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:40.021476 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:32:40.075820 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:40.026234 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 19:32:40.075820 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:40.026298 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" podUID="2123dbbb-db97-46f3-b265-a81414f0652c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 19:32:41.026057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:41.014027 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 19:32:41.026057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:41.014092 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" podUID="2123dbbb-db97-46f3-b265-a81414f0652c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 19:32:42.008396 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:42.008357 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 19:32:42.008597 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:42.008435 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" podUID="2123dbbb-db97-46f3-b265-a81414f0652c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 19:32:43.008154 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:43.008116 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 19:32:43.008574 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:43.008189 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" podUID="2123dbbb-db97-46f3-b265-a81414f0652c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 19:32:43.106931 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:43.106803 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:32:44.014527 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:44.014488 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 19:32:44.015169 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:44.015131 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" podUID="2123dbbb-db97-46f3-b265-a81414f0652c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 19:32:45.009547 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:45.009509 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 19:32:45.011184 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:45.011149 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" podUID="2123dbbb-db97-46f3-b265-a81414f0652c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 19:32:45.724046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:45.723958 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq"] Oct 30 19:32:45.740702 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:45.740669 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" Oct 30 19:32:45.771348 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:45.771295 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq"] Oct 30 19:32:45.786900 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:45.785060 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d9eb76\"" Oct 30 19:32:45.787155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:45.785161 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-d9eb76-dockercfg-sttl2\"" Oct 30 19:32:46.131057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:46.081283 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq\" (UID: \"4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" Oct 30 19:32:46.131057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:46.081698 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbgqn\" (UniqueName: \"kubernetes.io/projected/4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0-kube-api-access-wbgqn\") pod \"payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq\" (UID: \"4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" Oct 30 19:32:46.199050 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:46.198994 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq\" (UID: \"4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" Oct 30 19:32:46.208611 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:46.208572 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbgqn\" (UniqueName: \"kubernetes.io/projected/4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0-kube-api-access-wbgqn\") pod \"payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq\" (UID: \"4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" Oct 30 19:32:46.261467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:46.261419 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq\" (UID: \"4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" Oct 30 19:32:46.324459 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:46.324418 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:32:46.332667 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:46.332635 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:32:46.343323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:46.343289 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 19:32:46.367031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:46.354780 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbgqn\" (UniqueName: \"kubernetes.io/projected/4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0-kube-api-access-wbgqn\") pod \"payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq\" (UID: \"4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" Oct 30 19:32:46.404822 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:46.404183 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" Oct 30 19:32:48.293032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:48.242520 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 30 19:32:48.495512 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:32:48.274524 5047 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.208s" Oct 30 19:32:48.495512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:48.295068 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:48.495512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:48.381230 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:48.606790 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:48.566500 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:32:48.921998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:48.921891 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq"] Oct 30 19:32:50.083533 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:50.083480 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" event={"ID":"4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0","Type":"ContainerStarted","Data":"3928d8b75b01c118473b9989d7b2dddf74eec63440af3c0643f278b8dd4df711"} Oct 30 19:32:50.083967 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:50.083549 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" event={"ID":"4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0","Type":"ContainerStarted","Data":"c4a40bf35c96dfd70305d56d6303697f8647e74dd01542411c3238e1f9594e3d"} Oct 30 19:32:50.083967 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:50.083633 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" Oct 30 19:32:50.124323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:50.124265 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" podStartSLOduration=5.124249425 podStartE2EDuration="5.124249425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:50.120896043 +0000 UTC m=+1723.195351402" watchObservedRunningTime="2026-10-30 19:32:50.124249425 +0000 UTC m=+1723.198704821" Oct 30 19:32:56.444209 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:32:56.444170 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb"] Oct 30 19:33:01.096292 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:01.096260 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" Oct 30 19:33:01.444531 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:01.444415 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" podUID="d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f" containerName="payload-processing" containerID="cri-o://b87972331e46c4a7a981ab7debf43e9b3808214824b56a2ca406b1611534b001" gracePeriod=25 Oct 30 19:33:01.920038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:01.919944 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" Oct 30 19:33:02.064628 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:02.064522 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f-kube-api-access-5qj95\" (UniqueName: \"kubernetes.io/projected/d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f-kube-api-access-5qj95\") pod \"d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f\" (UID: \"d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f\") " Oct 30 19:33:02.064628 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:02.064588 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f-plugins-config-volume\") pod \"d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f\" (UID: \"d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f\") " Oct 30 19:33:02.065114 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:02.065082 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f-plugins-config-volume" pod "d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f" (UID: "d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:02.065533 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:02.065510 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:33:02.066904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:02.066871 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f-kube-api-access-5qj95" pod "d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f" (UID: "d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f"). InnerVolumeSpecName "kube-api-access-5qj95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:02.167385 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:02.167347 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qj95\" (UniqueName: \"kubernetes.io/projected/d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f-kube-api-access-5qj95\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:33:02.236048 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:02.235996 5047 generic.go:356] "Generic (PLEG): container finished" podID="d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f" containerID="b87972331e46c4a7a981ab7debf43e9b3808214824b56a2ca406b1611534b001" exitCode=0 Oct 30 19:33:02.236234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:02.236084 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" event={"ID":"d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f","Type":"ContainerDied","Data":"b87972331e46c4a7a981ab7debf43e9b3808214824b56a2ca406b1611534b001"} Oct 30 19:33:02.236234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:02.236094 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" Oct 30 19:33:02.236234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:02.236135 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb" event={"ID":"d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f","Type":"ContainerDied","Data":"90e80a72f54e75eb9a715d61cc1d03a392346cd67f11e3f1f85618e01fa526f1"} Oct 30 19:33:02.236234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:02.236157 5047 scope.go:122] "RemoveContainer" containerID="b87972331e46c4a7a981ab7debf43e9b3808214824b56a2ca406b1611534b001" Oct 30 19:33:02.254912 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:02.254888 5047 scope.go:122] "RemoveContainer" containerID="b87972331e46c4a7a981ab7debf43e9b3808214824b56a2ca406b1611534b001" Oct 30 19:33:02.255297 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:33:02.255263 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b87972331e46c4a7a981ab7debf43e9b3808214824b56a2ca406b1611534b001\": container with ID starting with b87972331e46c4a7a981ab7debf43e9b3808214824b56a2ca406b1611534b001 not found: ID does not exist" containerID="b87972331e46c4a7a981ab7debf43e9b3808214824b56a2ca406b1611534b001" Oct 30 19:33:02.255380 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:02.255301 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b87972331e46c4a7a981ab7debf43e9b3808214824b56a2ca406b1611534b001"} err="failed to get container status \"b87972331e46c4a7a981ab7debf43e9b3808214824b56a2ca406b1611534b001\": rpc error: code = NotFound desc = could not find container \"b87972331e46c4a7a981ab7debf43e9b3808214824b56a2ca406b1611534b001\": container with ID starting with b87972331e46c4a7a981ab7debf43e9b3808214824b56a2ca406b1611534b001 not found: ID does not exist" Oct 30 19:33:02.288607 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:02.288569 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb"] Oct 30 19:33:02.292927 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:02.292894 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-87b0dcce-777cd968dd-mkkrb"] Oct 30 19:33:03.051602 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:03.051569 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f" path="/var/lib/kubelet/pods/d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f/volumes" Oct 30 19:33:06.769594 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:06.769559 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw"] Oct 30 19:33:06.770380 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:06.770363 5047 cpu_manager.go:414] "containerMap: removing container" podUID="d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f" containerName="payload-processing" Oct 30 19:33:06.770380 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:06.770381 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f" containerName="payload-processing" Oct 30 19:33:06.770540 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:06.770531 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68f5e81-a8e6-4d59-9024-99f6a7f7ed0f" containerName="payload-processing" Oct 30 19:33:06.773878 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:06.773861 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:33:06.795047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:06.794996 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw"] Oct 30 19:33:06.802022 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:06.799131 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-891d6b-serving-cert\"" Oct 30 19:33:06.869551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:06.869507 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b64318a8-a330-4651-8b72-726f3b31fce3-openshift-service-ca\") pod \"maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw\" (UID: \"b64318a8-a330-4651-8b72-726f3b31fce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:33:06.869879 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:06.869851 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b64318a8-a330-4651-8b72-726f3b31fce3-maas-api-tls\") pod \"maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw\" (UID: \"b64318a8-a330-4651-8b72-726f3b31fce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:33:06.869974 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:06.869957 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntpx6\" (UniqueName: \"kubernetes.io/projected/b64318a8-a330-4651-8b72-726f3b31fce3-kube-api-access-ntpx6\") pod \"maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw\" (UID: \"b64318a8-a330-4651-8b72-726f3b31fce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:33:06.870118 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:06.870102 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b64318a8-a330-4651-8b72-726f3b31fce3-metrics-certs\") pod \"maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw\" (UID: \"b64318a8-a330-4651-8b72-726f3b31fce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:33:06.975419 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:06.975374 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b64318a8-a330-4651-8b72-726f3b31fce3-maas-api-tls\") pod \"maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw\" (UID: \"b64318a8-a330-4651-8b72-726f3b31fce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:33:06.981018 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:06.980966 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntpx6\" (UniqueName: \"kubernetes.io/projected/b64318a8-a330-4651-8b72-726f3b31fce3-kube-api-access-ntpx6\") pod \"maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw\" (UID: \"b64318a8-a330-4651-8b72-726f3b31fce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:33:06.985033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:06.981452 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b64318a8-a330-4651-8b72-726f3b31fce3-metrics-certs\") pod \"maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw\" (UID: \"b64318a8-a330-4651-8b72-726f3b31fce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:33:06.985387 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:06.985359 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b64318a8-a330-4651-8b72-726f3b31fce3-openshift-service-ca\") pod \"maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw\" (UID: \"b64318a8-a330-4651-8b72-726f3b31fce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:33:07.016676 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:07.007323 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b64318a8-a330-4651-8b72-726f3b31fce3-maas-api-tls\") pod \"maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw\" (UID: \"b64318a8-a330-4651-8b72-726f3b31fce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:33:07.016911 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:07.014784 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b64318a8-a330-4651-8b72-726f3b31fce3-openshift-service-ca\") pod \"maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw\" (UID: \"b64318a8-a330-4651-8b72-726f3b31fce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:33:07.045259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:07.045165 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntpx6\" (UniqueName: \"kubernetes.io/projected/b64318a8-a330-4651-8b72-726f3b31fce3-kube-api-access-ntpx6\") pod \"maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw\" (UID: \"b64318a8-a330-4651-8b72-726f3b31fce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:33:07.051662 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:07.051594 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b64318a8-a330-4651-8b72-726f3b31fce3-metrics-certs\") pod \"maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw\" (UID: \"b64318a8-a330-4651-8b72-726f3b31fce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:33:07.089030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:07.088110 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:33:07.489746 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:33:07.489698 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64318a8_a330_4651_8b72_726f3b31fce3.slice/crio-baca58a13057fe7e043630b26468b31a1811a01f9a778117ccd3f55c967eb7a8 WatchSource:0}: Error finding container baca58a13057fe7e043630b26468b31a1811a01f9a778117ccd3f55c967eb7a8: Status 404 returned error can't find the container with id baca58a13057fe7e043630b26468b31a1811a01f9a778117ccd3f55c967eb7a8 Oct 30 19:33:07.489857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:07.489807 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw"] Oct 30 19:33:08.351311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:08.348505 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" event={"ID":"b64318a8-a330-4651-8b72-726f3b31fce3","Type":"ContainerStarted","Data":"6c869e88f2b1517c71add4501fc9bd3936454c3279058f7b1aae1ff46b7a785a"} Oct 30 19:33:08.351311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:08.348599 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" event={"ID":"b64318a8-a330-4651-8b72-726f3b31fce3","Type":"ContainerStarted","Data":"baca58a13057fe7e043630b26468b31a1811a01f9a778117ccd3f55c967eb7a8"} Oct 30 19:33:08.351311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:08.349992 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:33:08.412035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:08.409076 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" podStartSLOduration=1.98447084 podStartE2EDuration="2.40905606s" totalImagesPullingTime="424.58522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:08.40410705 +0000 UTC m=+1741.478562410" watchObservedRunningTime="2026-10-30 19:33:08.40905606 +0000 UTC m=+1741.483511433" Oct 30 19:33:09.604038 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:33:09.603979 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:33:12.382207 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.382157 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm"] Oct 30 19:33:12.391467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.391405 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.453207 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.431846 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ccf8bd61\"/\"openshift-service-ca.crt\"" Oct 30 19:33:12.454063 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.437175 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ccf8bd61\"/\"e2e-lifecycle-model-ccf8bd61-kserve-self-signed-certs\"" Oct 30 19:33:12.454269 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.437886 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ccf8bd61\"/\"default-dockercfg-mz5z9\"" Oct 30 19:33:12.454380 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.437987 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ccf8bd61\"/\"kube-root-ca.crt\"" Oct 30 19:33:12.550082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.529289 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-home\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.550082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.529420 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c935589c-ed56-4012-85fd-b628a0e50980-tls-certs\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.550082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.529462 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hzxr\" (UniqueName: \"kubernetes.io/projected/c935589c-ed56-4012-85fd-b628a0e50980-kube-api-access-8hzxr\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.550082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.529553 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-tmp-dir\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.550082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.529600 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-model-cache\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.550082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.529655 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-dshm\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.653088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.649501 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c935589c-ed56-4012-85fd-b628a0e50980-tls-certs\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.653088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.649560 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hzxr\" (UniqueName: \"kubernetes.io/projected/c935589c-ed56-4012-85fd-b628a0e50980-kube-api-access-8hzxr\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.653088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.649638 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-tmp-dir\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.653088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.649675 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-model-cache\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.653088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.649720 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-dshm\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.653088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.650071 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-home\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.732202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.697023 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-tmp-dir\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.732202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.697685 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-model-cache\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.732202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.697732 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm"] Oct 30 19:33:12.732202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.707351 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-home\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.746025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.745897 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hzxr\" (UniqueName: \"kubernetes.io/projected/c935589c-ed56-4012-85fd-b628a0e50980-kube-api-access-8hzxr\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.772131 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.772052 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c935589c-ed56-4012-85fd-b628a0e50980-tls-certs\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:12.805924 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:12.805854 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-dshm\") pod \"e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:13.004077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:13.000354 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:13.389324 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:13.389289 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm"] Oct 30 19:33:13.700091 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:13.695968 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" event={"ID":"c935589c-ed56-4012-85fd-b628a0e50980","Type":"ContainerStarted","Data":"8ec4f77584ac576d026a77e8d6a8ca3b0e6764d09b58187433bafb517529680a"} Oct 30 19:33:13.700091 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:13.696045 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" event={"ID":"c935589c-ed56-4012-85fd-b628a0e50980","Type":"ContainerStarted","Data":"f3176fdc853345bf35cf46374290676e44765b692ef74d8f16f643e7d52b2408"} Oct 30 19:33:13.753027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:13.752644 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" podStartSLOduration=1.752609311 podStartE2EDuration="1.752609311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:13.749996658 +0000 UTC m=+1746.824452018" watchObservedRunningTime="2026-10-30 19:33:13.752609311 +0000 UTC m=+1746.827064675" Oct 30 19:33:14.016179 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:14.013715 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:15.482650 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:15.482613 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:33:15.541152 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:15.541110 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": context deadline exceeded" Oct 30 19:33:15.756772 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:15.756666 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-s2nd8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:33:15.757076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:15.757042 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" podUID="074a1938-2380-4eb4-adf4-a15e3162507f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:15.813030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:15.804400 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:15.831367 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:15.830938 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-frwdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:33:15.831367 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:15.831064 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" podUID="5fcb783a-2210-41a3-8379-b2c417ac94d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:16.004879 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:15.987713 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:16.004879 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:15.991751 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 19:33:16.625092 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:16.624644 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-r9r2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:33:16.625654 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:16.625455 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" podUID="3824e276-5eec-4cea-a06b-c81cc9853ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:17.692113 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:17.692074 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm"] Oct 30 19:33:17.692584 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:17.692327 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" podUID="8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d" containerName="maas-api" containerID="cri-o://66dc7c192bbd5b2f7fc6abd79400c577683e4db7d06df7eb28dd6ea6464cc321" gracePeriod=30 Oct 30 19:33:18.228985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.228951 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:33:18.320130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.314565 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-openshift-service-ca\") pod \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\" (UID: \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\") " Oct 30 19:33:18.320130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.314715 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-kube-api-access-b4kvl\" (UniqueName: \"kubernetes.io/projected/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-kube-api-access-b4kvl\") pod \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\" (UID: \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\") " Oct 30 19:33:18.320130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.314822 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-metrics-certs\") pod \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\" (UID: \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\") " Oct 30 19:33:18.320130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.315072 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-maas-api-tls\") pod \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\" (UID: \"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d\") " Oct 30 19:33:18.322089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.320815 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-openshift-service-ca" pod "8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d" (UID: "8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:18.326983 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.326635 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-kube-api-access-b4kvl" pod "8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d" (UID: "8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d"). InnerVolumeSpecName "kube-api-access-b4kvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:18.330576 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.330487 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-metrics-certs" pod "8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d" (UID: "8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:33:18.349362 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.349315 5047 generic.go:356] "Generic (PLEG): container finished" podID="8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d" containerID="66dc7c192bbd5b2f7fc6abd79400c577683e4db7d06df7eb28dd6ea6464cc321" exitCode=0 Oct 30 19:33:18.349666 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.349631 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" event={"ID":"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d","Type":"ContainerDied","Data":"66dc7c192bbd5b2f7fc6abd79400c577683e4db7d06df7eb28dd6ea6464cc321"} Oct 30 19:33:18.349802 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.349779 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" event={"ID":"8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d","Type":"ContainerDied","Data":"c2cca957c00a45114f01aebcb5ad1b6d238ede12deb5af031782f053ad9f46f2"} Oct 30 19:33:18.349877 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.349827 5047 scope.go:122] "RemoveContainer" containerID="66dc7c192bbd5b2f7fc6abd79400c577683e4db7d06df7eb28dd6ea6464cc321" Oct 30 19:33:18.350083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.350061 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm" Oct 30 19:33:18.362565 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.362504 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-maas-api-tls" pod "8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d" (UID: "8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:33:18.426054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.418338 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:33:18.426054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.418377 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:33:18.426054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.418459 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4kvl\" (UniqueName: \"kubernetes.io/projected/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-kube-api-access-b4kvl\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:33:18.426054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.418475 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:33:18.619233 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.619197 5047 scope.go:122] "RemoveContainer" containerID="66dc7c192bbd5b2f7fc6abd79400c577683e4db7d06df7eb28dd6ea6464cc321" Oct 30 19:33:18.678772 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:33:18.678720 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66dc7c192bbd5b2f7fc6abd79400c577683e4db7d06df7eb28dd6ea6464cc321\": container with ID starting with 66dc7c192bbd5b2f7fc6abd79400c577683e4db7d06df7eb28dd6ea6464cc321 not found: ID does not exist" containerID="66dc7c192bbd5b2f7fc6abd79400c577683e4db7d06df7eb28dd6ea6464cc321" Oct 30 19:33:18.678944 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.678787 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66dc7c192bbd5b2f7fc6abd79400c577683e4db7d06df7eb28dd6ea6464cc321"} err="failed to get container status \"66dc7c192bbd5b2f7fc6abd79400c577683e4db7d06df7eb28dd6ea6464cc321\": rpc error: code = NotFound desc = could not find container \"66dc7c192bbd5b2f7fc6abd79400c577683e4db7d06df7eb28dd6ea6464cc321\": container with ID starting with 66dc7c192bbd5b2f7fc6abd79400c577683e4db7d06df7eb28dd6ea6464cc321 not found: ID does not exist" Oct 30 19:33:18.741580 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.741544 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm"] Oct 30 19:33:18.750025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:18.749966 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-87b0dcce-7bf95d6cb7-qk6jm"] Oct 30 19:33:19.052176 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:19.052097 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d" path="/var/lib/kubelet/pods/8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d/volumes" Oct 30 19:33:20.488135 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.488097 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf"] Oct 30 19:33:20.488950 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.488933 5047 cpu_manager.go:414] "containerMap: removing container" podUID="8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d" containerName="maas-api" Oct 30 19:33:20.489015 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.488950 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d" containerName="maas-api" Oct 30 19:33:20.489088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.489078 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee0d7e9-4541-4829-8ef3-cbb8c2f2b50d" containerName="maas-api" Oct 30 19:33:20.494379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.494353 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 19:33:20.515975 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.515932 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf"] Oct 30 19:33:20.520398 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.520369 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-cd4bde-serving-cert\"" Oct 30 19:33:20.621913 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.621861 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fst6n\" (UniqueName: \"kubernetes.io/projected/874ca58f-1ef6-4395-bc98-892403898cf0-kube-api-access-fst6n\") pod \"maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf\" (UID: \"874ca58f-1ef6-4395-bc98-892403898cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 19:33:20.622199 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.621918 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/874ca58f-1ef6-4395-bc98-892403898cf0-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf\" (UID: \"874ca58f-1ef6-4395-bc98-892403898cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 19:33:20.622199 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.622140 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/874ca58f-1ef6-4395-bc98-892403898cf0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf\" (UID: \"874ca58f-1ef6-4395-bc98-892403898cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 19:33:20.622330 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.622251 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/874ca58f-1ef6-4395-bc98-892403898cf0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf\" (UID: \"874ca58f-1ef6-4395-bc98-892403898cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 19:33:20.728062 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.727336 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/874ca58f-1ef6-4395-bc98-892403898cf0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf\" (UID: \"874ca58f-1ef6-4395-bc98-892403898cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 19:33:20.728062 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.727511 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/874ca58f-1ef6-4395-bc98-892403898cf0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf\" (UID: \"874ca58f-1ef6-4395-bc98-892403898cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 19:33:20.728062 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.727767 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fst6n\" (UniqueName: \"kubernetes.io/projected/874ca58f-1ef6-4395-bc98-892403898cf0-kube-api-access-fst6n\") pod \"maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf\" (UID: \"874ca58f-1ef6-4395-bc98-892403898cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 19:33:20.728062 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.727813 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/874ca58f-1ef6-4395-bc98-892403898cf0-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf\" (UID: \"874ca58f-1ef6-4395-bc98-892403898cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 19:33:20.737744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.729080 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/874ca58f-1ef6-4395-bc98-892403898cf0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf\" (UID: \"874ca58f-1ef6-4395-bc98-892403898cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 19:33:20.761963 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.760653 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fst6n\" (UniqueName: \"kubernetes.io/projected/874ca58f-1ef6-4395-bc98-892403898cf0-kube-api-access-fst6n\") pod \"maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf\" (UID: \"874ca58f-1ef6-4395-bc98-892403898cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 19:33:20.766561 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.762398 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/874ca58f-1ef6-4395-bc98-892403898cf0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf\" (UID: \"874ca58f-1ef6-4395-bc98-892403898cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 19:33:20.766561 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.764899 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/874ca58f-1ef6-4395-bc98-892403898cf0-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf\" (UID: \"874ca58f-1ef6-4395-bc98-892403898cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 19:33:20.814030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:20.813380 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 19:33:21.177923 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:33:21.177891 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874ca58f_1ef6_4395_bc98_892403898cf0.slice/crio-fba2e884ec08862031196962ca95e9cb9694af541e4529970f54736f0ebe41a1 WatchSource:0}: Error finding container fba2e884ec08862031196962ca95e9cb9694af541e4529970f54736f0ebe41a1: Status 404 returned error can't find the container with id fba2e884ec08862031196962ca95e9cb9694af541e4529970f54736f0ebe41a1 Oct 30 19:33:21.180359 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:21.180250 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf"] Oct 30 19:33:21.404028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:21.403397 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" event={"ID":"874ca58f-1ef6-4395-bc98-892403898cf0","Type":"ContainerStarted","Data":"fba2e884ec08862031196962ca95e9cb9694af541e4529970f54736f0ebe41a1"} Oct 30 19:33:22.413195 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:22.413156 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" event={"ID":"874ca58f-1ef6-4395-bc98-892403898cf0","Type":"ContainerStarted","Data":"83513bec1c8b847ef7b053866c76211584331c4be1f53958c8365bc953ac2529"} Oct 30 19:33:22.413630 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:22.413367 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 19:33:22.459529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:22.459480 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" podStartSLOduration=1.985607586 podStartE2EDuration="2.45946646s" totalImagesPullingTime="473.858874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:22.459319494 +0000 UTC m=+1755.533774866" watchObservedRunningTime="2026-10-30 19:33:22.45946646 +0000 UTC m=+1755.533921820" Oct 30 19:33:23.001429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:23.001387 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:23.014574 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:23.014545 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:23.431629 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:23.431600 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:33:28.426592 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:28.426560 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 19:33:39.767918 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:39.752501 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt"] Oct 30 19:33:39.778248 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:33:39.778139 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:33:39.799839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:39.797762 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt"] Oct 30 19:33:39.799839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:39.797900 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:39.836127 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:39.835945 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-d9eb76-openshift-default-dockercfg-6vnqr\"" Oct 30 19:33:39.897850 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:39.897811 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:39.898110 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:39.897910 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:39.898184 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:39.898123 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfef2562-28c6-4640-a1c0-8a467bba8a79-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:39.898320 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:39.898292 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hffxn\" (UniqueName: \"kubernetes.io/projected/bfef2562-28c6-4640-a1c0-8a467bba8a79-kube-api-access-hffxn\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:39.898379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:39.898364 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:39.898506 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:39.898490 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:39.898561 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:39.898539 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:39.898609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:39.898574 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:39.900242 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:39.900219 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.002228 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.002174 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.002422 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.002243 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.002422 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.002338 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfef2562-28c6-4640-a1c0-8a467bba8a79-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.002537 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.002446 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hffxn\" (UniqueName: \"kubernetes.io/projected/bfef2562-28c6-4640-a1c0-8a467bba8a79-kube-api-access-hffxn\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.002537 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.002502 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.002641 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.002564 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.002641 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.002606 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.002752 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.002640 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.002752 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.002686 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.002752 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.002708 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.003054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.003027 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.003214 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.003093 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.003295 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.003220 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfef2562-28c6-4640-a1c0-8a467bba8a79-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.003463 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.003435 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.004905 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.004883 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.006073 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.006049 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.021904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.021831 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hffxn\" (UniqueName: \"kubernetes.io/projected/bfef2562-28c6-4640-a1c0-8a467bba8a79-kube-api-access-hffxn\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.022081 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.021943 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.125715 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.125669 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:40.600429 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:33:40.600393 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfef2562_28c6_4640_a1c0_8a467bba8a79.slice/crio-36e2a53bfc4caeaa8dbcad2569619694e9d9a2a5ad2de498997d85cc29b8ac37 WatchSource:0}: Error finding container 36e2a53bfc4caeaa8dbcad2569619694e9d9a2a5ad2de498997d85cc29b8ac37: Status 404 returned error can't find the container with id 36e2a53bfc4caeaa8dbcad2569619694e9d9a2a5ad2de498997d85cc29b8ac37 Oct 30 19:33:40.601668 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.601641 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt"] Oct 30 19:33:40.604180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.604146 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:33:40.604303 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.604278 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:33:40.604353 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.604336 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:33:40.806034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.805696 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" event={"ID":"bfef2562-28c6-4640-a1c0-8a467bba8a79","Type":"ContainerStarted","Data":"5c6cea449c1c2678df502ac0ed48cc5fe496d47722d4f18429973ea3e2107f3e"} Oct 30 19:33:40.806034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.805761 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" event={"ID":"bfef2562-28c6-4640-a1c0-8a467bba8a79","Type":"ContainerStarted","Data":"36e2a53bfc4caeaa8dbcad2569619694e9d9a2a5ad2de498997d85cc29b8ac37"} Oct 30 19:33:40.859624 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:40.859499 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" podStartSLOduration=1.8594779529999999 podStartE2EDuration="1.859477953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:40.855104796 +0000 UTC m=+1773.929560156" watchObservedRunningTime="2026-10-30 19:33:40.859477953 +0000 UTC m=+1773.933933313" Oct 30 19:33:41.125908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:41.125811 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:41.127450 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:41.127390 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:33:41.127599 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:41.127448 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" podUID="bfef2562-28c6-4640-a1c0-8a467bba8a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 19:33:42.265437 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:42.251735 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:33:42.265437 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:42.251803 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" podUID="bfef2562-28c6-4640-a1c0-8a467bba8a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 19:33:43.126967 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:43.126927 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:33:43.127222 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:43.127058 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" podUID="bfef2562-28c6-4640-a1c0-8a467bba8a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 19:33:43.989647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:43.989555 5047 generic.go:356] "Generic (PLEG): container finished" podID="faec438d-9b43-450a-b39e-e6eabb6e0b98" containerID="d649a5636675450846e4a3effd0c5f3d2c9efd4b5b1d444076e3d99ee5b21059" exitCode=0 Oct 30 19:33:43.989647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:43.989608 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" event={"ID":"faec438d-9b43-450a-b39e-e6eabb6e0b98","Type":"ContainerDied","Data":"d649a5636675450846e4a3effd0c5f3d2c9efd4b5b1d444076e3d99ee5b21059"} Oct 30 19:33:43.990166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:43.990151 5047 scope.go:122] "RemoveContainer" containerID="d649a5636675450846e4a3effd0c5f3d2c9efd4b5b1d444076e3d99ee5b21059" Oct 30 19:33:43.991854 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:43.991829 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:33:43.991915 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:43.991896 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:33:43.991953 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:43.991930 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:33:44.126527 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:44.126494 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:33:44.126651 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:44.126555 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" podUID="bfef2562-28c6-4640-a1c0-8a467bba8a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 19:33:44.999053 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:44.999016 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" event={"ID":"faec438d-9b43-450a-b39e-e6eabb6e0b98","Type":"ContainerStarted","Data":"62a8abd4e9dbd99351780fc8efd3fcb0d6cb4bdadd99c173eac5fde05bad7d73"} Oct 30 19:33:45.126253 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:45.126218 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:33:45.126451 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:45.126288 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" podUID="bfef2562-28c6-4640-a1c0-8a467bba8a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 19:33:45.951025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:45.950241 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:33:45.955028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:45.951464 5047 patch_prober.go:29] interesting pod/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:33:45.955028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:45.951572 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" podUID="faec438d-9b43-450a-b39e-e6eabb6e0b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:33:46.176032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:46.155038 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:33:46.176032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:46.155146 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" podUID="bfef2562-28c6-4640-a1c0-8a467bba8a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 19:33:46.951031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:46.950444 5047 patch_prober.go:29] interesting pod/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:33:46.951031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:46.950512 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" podUID="faec438d-9b43-450a-b39e-e6eabb6e0b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:33:47.010577 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:47.010504 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:33:47.126614 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:47.126573 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:33:47.126793 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:47.126637 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" podUID="bfef2562-28c6-4640-a1c0-8a467bba8a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 19:33:47.959689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:47.959641 5047 patch_prober.go:29] interesting pod/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:33:47.960246 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:47.959707 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" podUID="faec438d-9b43-450a-b39e-e6eabb6e0b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:33:48.131601 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:48.131570 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:48.132458 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:48.132435 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:48.133374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:48.133356 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 19:33:48.946444 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:48.946407 5047 patch_prober.go:29] interesting pod/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:33:48.946724 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:48.946695 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" podUID="faec438d-9b43-450a-b39e-e6eabb6e0b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:33:49.943027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:49.942964 5047 patch_prober.go:29] interesting pod/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:33:49.943705 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:49.943669 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" podUID="faec438d-9b43-450a-b39e-e6eabb6e0b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:33:50.950036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:50.944416 5047 patch_prober.go:29] interesting pod/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:33:50.950036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:50.944522 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" podUID="faec438d-9b43-450a-b39e-e6eabb6e0b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:33:51.947812 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:51.947776 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:33:51.949085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:33:51.949061 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:34:01.451462 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.451416 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9"] Oct 30 19:34:01.456333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.456302 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.471476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.471444 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f4ab4a-openshift-default-dockercfg-hwh58\"" Oct 30 19:34:01.481331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.478165 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-credential-socket\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.481331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.478353 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-envoy\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.481331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.478397 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1713eb56-d041-40ae-824a-3bdc1ca843f9-istiod-ca-cert\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.481331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.478562 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-data\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.481331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.478599 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmmsp\" (UniqueName: \"kubernetes.io/projected/1713eb56-d041-40ae-824a-3bdc1ca843f9-kube-api-access-hmmsp\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.481331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.478639 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-podinfo\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.481331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.478717 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-token\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.481331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.478747 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-workload-certs\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.481331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.478786 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-workload-socket\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.494492 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.494455 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9"] Oct 30 19:34:01.587432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.584496 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-envoy\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.587432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.584564 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1713eb56-d041-40ae-824a-3bdc1ca843f9-istiod-ca-cert\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.587432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.584684 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-data\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.587432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.584739 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmmsp\" (UniqueName: \"kubernetes.io/projected/1713eb56-d041-40ae-824a-3bdc1ca843f9-kube-api-access-hmmsp\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.587432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.584870 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-podinfo\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.587432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.585020 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-token\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.587432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.585060 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-workload-certs\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.587432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.585119 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-workload-socket\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.587432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.585307 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-credential-socket\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.587432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.585939 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-credential-socket\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.587432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.586732 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-workload-certs\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.587432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.587340 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-workload-socket\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.613033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.608493 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1713eb56-d041-40ae-824a-3bdc1ca843f9-istiod-ca-cert\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.649030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.613668 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-data\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.727082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.720777 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-envoy\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.764905 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.764852 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmmsp\" (UniqueName: \"kubernetes.io/projected/1713eb56-d041-40ae-824a-3bdc1ca843f9-kube-api-access-hmmsp\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.806247 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.803817 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-token\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.806247 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.803942 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-podinfo\") pod \"e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:01.889400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.889079 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd"] Oct 30 19:34:01.890514 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.890478 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" podUID="9e15d741-48fd-4962-90db-de0e6cb664f3" containerName="istio-proxy" containerID="cri-o://7a12cf1a6ddf89b7ea2eec687c46087565bf565148655daeefb226bf951ad56d" gracePeriod=30 Oct 30 19:34:01.989037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.965989 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg"] Oct 30 19:34:01.989037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:01.966368 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" podUID="bccbf42c-7950-485f-a5b4-0475f827c419" containerName="istio-proxy" containerID="cri-o://be8cd1ecb2db0cc524b9c5aad84833f17c7d32edd9222b650338660bf689cb4c" gracePeriod=30 Oct 30 19:34:02.123598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:02.123555 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5"] Oct 30 19:34:02.123808 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:02.123645 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95"] Oct 30 19:34:02.125309 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:02.125266 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:02.133369 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:02.133295 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" podUID="708f310c-157e-4c22-8de1-1bdecacc2023" containerName="istio-proxy" containerID="cri-o://fec855be18ec33febbacea02dd182403570d82b24bbd4abaa97d9c1c7868e056" gracePeriod=30 Oct 30 19:34:02.150037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:02.149953 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" podUID="586ca704-ecaf-4d51-88b4-6ea6770bb10f" containerName="istio-proxy" containerID="cri-o://e47cd3dfe05e82d988c03f5cecafe8c7b4ff400a2c4883f34dbd29dad24639b3" gracePeriod=30 Oct 30 19:34:02.683271 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:34:02.683232 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1713eb56_d041_40ae_824a_3bdc1ca843f9.slice/crio-a527ab04574f6b5c3d20467878569258fbc0916c670c8faa6f618201f5ff21ba WatchSource:0}: Error finding container a527ab04574f6b5c3d20467878569258fbc0916c670c8faa6f618201f5ff21ba: Status 404 returned error can't find the container with id a527ab04574f6b5c3d20467878569258fbc0916c670c8faa6f618201f5ff21ba Oct 30 19:34:02.686082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:02.685998 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9"] Oct 30 19:34:02.686990 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:02.686945 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:02.687117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:02.687059 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:02.687158 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:02.687115 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:03.425192 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:03.425154 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" event={"ID":"1713eb56-d041-40ae-824a-3bdc1ca843f9","Type":"ContainerStarted","Data":"501f90f756a7578030e096d0b1936d121121055b50a97aff3e43ae496d9a223f"} Oct 30 19:34:03.425409 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:03.425211 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" event={"ID":"1713eb56-d041-40ae-824a-3bdc1ca843f9","Type":"ContainerStarted","Data":"a527ab04574f6b5c3d20467878569258fbc0916c670c8faa6f618201f5ff21ba"} Oct 30 19:34:03.481588 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:03.481526 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" podStartSLOduration=2.481510001 podStartE2EDuration="2.481510001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:03.477124351 +0000 UTC m=+1796.551579712" watchObservedRunningTime="2026-10-30 19:34:03.481510001 +0000 UTC m=+1796.555965360" Oct 30 19:34:04.150473 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:04.150436 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:04.156405 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:04.156374 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:04.433823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:04.433734 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:04.434712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:04.434691 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:34:05.435652 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.435617 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2"] Oct 30 19:34:05.441306 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.441277 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.460455 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.460425 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2"] Oct 30 19:34:05.476938 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.476903 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7d9e82-openshift-default-dockercfg-w9n54\"" Oct 30 19:34:05.612069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.582407 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-token\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.612069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.582479 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-workload-certs\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.612069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.582550 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-credential-socket\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.612069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.582655 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-workload-socket\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.612069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.582836 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-data\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.612069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.582990 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-envoy\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.612069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.583083 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istiod-ca-cert\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.612069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.583113 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-podinfo\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.612069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.583137 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdkp7\" (UniqueName: \"kubernetes.io/projected/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-kube-api-access-sdkp7\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.741529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.696566 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-envoy\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.741529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.696666 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istiod-ca-cert\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.741529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.696693 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-podinfo\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.741529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.696714 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdkp7\" (UniqueName: \"kubernetes.io/projected/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-kube-api-access-sdkp7\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.741529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.696770 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-token\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.741529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.696818 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-workload-certs\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.741529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.696887 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-credential-socket\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.741529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.697031 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-workload-socket\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.741529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.697236 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-data\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.741529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.698357 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-credential-socket\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.741529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.698667 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-workload-socket\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.741529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.698880 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-data\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.741529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.699452 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-workload-certs\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.741529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.699456 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istiod-ca-cert\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.741529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.719753 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-envoy\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.743236 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.743203 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-podinfo\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.762500 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.759258 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdkp7\" (UniqueName: \"kubernetes.io/projected/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-kube-api-access-sdkp7\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.762758 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.762693 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-token\") pod \"e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:05.779557 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:05.779508 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:06.650580 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:06.641606 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2"] Oct 30 19:34:06.659050 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:34:06.658743 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b488a6e_9b1c_44c0_8656_0bd32d3e3b96.slice/crio-061bcb80d1fe10fdc4a7ab1787ec9b92aaa156ea67371696402058349170667e WatchSource:0}: Error finding container 061bcb80d1fe10fdc4a7ab1787ec9b92aaa156ea67371696402058349170667e: Status 404 returned error can't find the container with id 061bcb80d1fe10fdc4a7ab1787ec9b92aaa156ea67371696402058349170667e Oct 30 19:34:06.671425 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:06.671049 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:06.671854 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:06.671828 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:06.672030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:06.671993 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:07.253885 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.253852 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:34:07.350130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.350088 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e15d741-48fd-4962-90db-de0e6cb664f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e15d741-48fd-4962-90db-de0e6cb664f3-istiod-ca-cert\") pod \"9e15d741-48fd-4962-90db-de0e6cb664f3\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " Oct 30 19:34:07.350325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.350188 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-workload-certs\") pod \"9e15d741-48fd-4962-90db-de0e6cb664f3\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " Oct 30 19:34:07.350325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.350280 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-data\") pod \"9e15d741-48fd-4962-90db-de0e6cb664f3\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " Oct 30 19:34:07.350439 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.350383 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-podinfo\") pod \"9e15d741-48fd-4962-90db-de0e6cb664f3\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " Oct 30 19:34:07.350439 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.350427 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e15d741-48fd-4962-90db-de0e6cb664f3-kube-api-access-8zftl\" (UniqueName: \"kubernetes.io/projected/9e15d741-48fd-4962-90db-de0e6cb664f3-kube-api-access-8zftl\") pod \"9e15d741-48fd-4962-90db-de0e6cb664f3\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " Oct 30 19:34:07.350568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.350550 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-workload-socket\") pod \"9e15d741-48fd-4962-90db-de0e6cb664f3\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " Oct 30 19:34:07.350685 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.350669 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-envoy\") pod \"9e15d741-48fd-4962-90db-de0e6cb664f3\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " Oct 30 19:34:07.350831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.350813 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-token\" (UniqueName: \"kubernetes.io/projected/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-token\") pod \"9e15d741-48fd-4962-90db-de0e6cb664f3\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " Oct 30 19:34:07.350914 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.350899 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-credential-socket\") pod \"9e15d741-48fd-4962-90db-de0e6cb664f3\" (UID: \"9e15d741-48fd-4962-90db-de0e6cb664f3\") " Oct 30 19:34:07.352531 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.352326 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-workload-certs" pod "9e15d741-48fd-4962-90db-de0e6cb664f3" (UID: "9e15d741-48fd-4962-90db-de0e6cb664f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.354355 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.354170 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-podinfo" pod "9e15d741-48fd-4962-90db-de0e6cb664f3" (UID: "9e15d741-48fd-4962-90db-de0e6cb664f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:34:07.354664 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.354532 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-data" pod "9e15d741-48fd-4962-90db-de0e6cb664f3" (UID: "9e15d741-48fd-4962-90db-de0e6cb664f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.355068 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.354777 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-workload-socket" pod "9e15d741-48fd-4962-90db-de0e6cb664f3" (UID: "9e15d741-48fd-4962-90db-de0e6cb664f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.359082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.359048 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e15d741-48fd-4962-90db-de0e6cb664f3-istiod-ca-cert" pod "9e15d741-48fd-4962-90db-de0e6cb664f3" (UID: "9e15d741-48fd-4962-90db-de0e6cb664f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:07.360892 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.360730 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e15d741-48fd-4962-90db-de0e6cb664f3-kube-api-access-8zftl" pod "9e15d741-48fd-4962-90db-de0e6cb664f3" (UID: "9e15d741-48fd-4962-90db-de0e6cb664f3"). InnerVolumeSpecName "kube-api-access-8zftl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:07.362227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.362171 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-credential-socket" pod "9e15d741-48fd-4962-90db-de0e6cb664f3" (UID: "9e15d741-48fd-4962-90db-de0e6cb664f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.364552 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.364467 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-token" pod "9e15d741-48fd-4962-90db-de0e6cb664f3" (UID: "9e15d741-48fd-4962-90db-de0e6cb664f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:07.365307 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.365274 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-envoy" pod "9e15d741-48fd-4962-90db-de0e6cb664f3" (UID: "9e15d741-48fd-4962-90db-de0e6cb664f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.366968 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.366931 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.367115 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.366977 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zftl\" (UniqueName: \"kubernetes.io/projected/9e15d741-48fd-4962-90db-de0e6cb664f3-kube-api-access-8zftl\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.367115 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.366997 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.367115 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.367070 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.367115 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.367086 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.367115 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.367099 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.367115 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.367113 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e15d741-48fd-4962-90db-de0e6cb664f3-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.367400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.367128 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.367400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.367144 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e15d741-48fd-4962-90db-de0e6cb664f3-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.527533 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.527501 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:34:07.538877 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.538842 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:34:07.542537 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.542507 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:34:07.555710 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.555675 5047 generic.go:356] "Generic (PLEG): container finished" podID="586ca704-ecaf-4d51-88b4-6ea6770bb10f" containerID="e47cd3dfe05e82d988c03f5cecafe8c7b4ff400a2c4883f34dbd29dad24639b3" exitCode=0 Oct 30 19:34:07.555896 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.555780 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" event={"ID":"586ca704-ecaf-4d51-88b4-6ea6770bb10f","Type":"ContainerDied","Data":"e47cd3dfe05e82d988c03f5cecafe8c7b4ff400a2c4883f34dbd29dad24639b3"} Oct 30 19:34:07.558076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.557996 5047 generic.go:356] "Generic (PLEG): container finished" podID="bccbf42c-7950-485f-a5b4-0475f827c419" containerID="be8cd1ecb2db0cc524b9c5aad84833f17c7d32edd9222b650338660bf689cb4c" exitCode=0 Oct 30 19:34:07.558202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.558149 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" event={"ID":"bccbf42c-7950-485f-a5b4-0475f827c419","Type":"ContainerDied","Data":"be8cd1ecb2db0cc524b9c5aad84833f17c7d32edd9222b650338660bf689cb4c"} Oct 30 19:34:07.560458 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.560438 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:34:07.560876 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.560857 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" event={"ID":"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96","Type":"ContainerStarted","Data":"ee8d8c04f7a511e00231382c4cbba30d2e7ff086f1260edcd3dfe2016dc256db"} Oct 30 19:34:07.560952 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.560909 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" event={"ID":"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96","Type":"ContainerStarted","Data":"061bcb80d1fe10fdc4a7ab1787ec9b92aaa156ea67371696402058349170667e"} Oct 30 19:34:07.562391 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.562365 5047 generic.go:356] "Generic (PLEG): container finished" podID="708f310c-157e-4c22-8de1-1bdecacc2023" containerID="fec855be18ec33febbacea02dd182403570d82b24bbd4abaa97d9c1c7868e056" exitCode=0 Oct 30 19:34:07.562468 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.562396 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" Oct 30 19:34:07.562468 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.562435 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" event={"ID":"708f310c-157e-4c22-8de1-1bdecacc2023","Type":"ContainerDied","Data":"fec855be18ec33febbacea02dd182403570d82b24bbd4abaa97d9c1c7868e056"} Oct 30 19:34:07.562547 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.562478 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95" event={"ID":"708f310c-157e-4c22-8de1-1bdecacc2023","Type":"ContainerDied","Data":"20d3a3d7259b4bc0e2eba23d076670d742646ee54aba4a80fb3b214c29ccf3f4"} Oct 30 19:34:07.562547 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.562503 5047 scope.go:122] "RemoveContainer" containerID="fec855be18ec33febbacea02dd182403570d82b24bbd4abaa97d9c1c7868e056" Oct 30 19:34:07.564663 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.564645 5047 generic.go:356] "Generic (PLEG): container finished" podID="9e15d741-48fd-4962-90db-de0e6cb664f3" containerID="7a12cf1a6ddf89b7ea2eec687c46087565bf565148655daeefb226bf951ad56d" exitCode=0 Oct 30 19:34:07.564758 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.564690 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" Oct 30 19:34:07.564758 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.564727 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" event={"ID":"9e15d741-48fd-4962-90db-de0e6cb664f3","Type":"ContainerDied","Data":"7a12cf1a6ddf89b7ea2eec687c46087565bf565148655daeefb226bf951ad56d"} Oct 30 19:34:07.564834 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.564782 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd" event={"ID":"9e15d741-48fd-4962-90db-de0e6cb664f3","Type":"ContainerDied","Data":"5ec5ecdafcd62554dee89f418e7ae4d08b1755a075eb48b8b603c3e1d47cbcad"} Oct 30 19:34:07.595378 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.594155 5047 scope.go:122] "RemoveContainer" containerID="fec855be18ec33febbacea02dd182403570d82b24bbd4abaa97d9c1c7868e056" Oct 30 19:34:07.595378 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:34:07.594642 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec855be18ec33febbacea02dd182403570d82b24bbd4abaa97d9c1c7868e056\": container with ID starting with fec855be18ec33febbacea02dd182403570d82b24bbd4abaa97d9c1c7868e056 not found: ID does not exist" containerID="fec855be18ec33febbacea02dd182403570d82b24bbd4abaa97d9c1c7868e056" Oct 30 19:34:07.595378 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.594686 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec855be18ec33febbacea02dd182403570d82b24bbd4abaa97d9c1c7868e056"} err="failed to get container status \"fec855be18ec33febbacea02dd182403570d82b24bbd4abaa97d9c1c7868e056\": rpc error: code = NotFound desc = could not find container \"fec855be18ec33febbacea02dd182403570d82b24bbd4abaa97d9c1c7868e056\": container with ID starting with fec855be18ec33febbacea02dd182403570d82b24bbd4abaa97d9c1c7868e056 not found: ID does not exist" Oct 30 19:34:07.595378 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.594711 5047 scope.go:122] "RemoveContainer" containerID="7a12cf1a6ddf89b7ea2eec687c46087565bf565148655daeefb226bf951ad56d" Oct 30 19:34:07.619333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.619308 5047 scope.go:122] "RemoveContainer" containerID="7b4274f5e71fc6a82355b76c97e5554c897e19ce89d3a0c939945db10ff1e3e2" Oct 30 19:34:07.634482 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.634454 5047 scope.go:122] "RemoveContainer" containerID="7a12cf1a6ddf89b7ea2eec687c46087565bf565148655daeefb226bf951ad56d" Oct 30 19:34:07.634957 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:34:07.634926 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a12cf1a6ddf89b7ea2eec687c46087565bf565148655daeefb226bf951ad56d\": container with ID starting with 7a12cf1a6ddf89b7ea2eec687c46087565bf565148655daeefb226bf951ad56d not found: ID does not exist" containerID="7a12cf1a6ddf89b7ea2eec687c46087565bf565148655daeefb226bf951ad56d" Oct 30 19:34:07.635067 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.634971 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a12cf1a6ddf89b7ea2eec687c46087565bf565148655daeefb226bf951ad56d"} err="failed to get container status \"7a12cf1a6ddf89b7ea2eec687c46087565bf565148655daeefb226bf951ad56d\": rpc error: code = NotFound desc = could not find container \"7a12cf1a6ddf89b7ea2eec687c46087565bf565148655daeefb226bf951ad56d\": container with ID starting with 7a12cf1a6ddf89b7ea2eec687c46087565bf565148655daeefb226bf951ad56d not found: ID does not exist" Oct 30 19:34:07.635067 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.634998 5047 scope.go:122] "RemoveContainer" containerID="7b4274f5e71fc6a82355b76c97e5554c897e19ce89d3a0c939945db10ff1e3e2" Oct 30 19:34:07.635602 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:34:07.635578 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b4274f5e71fc6a82355b76c97e5554c897e19ce89d3a0c939945db10ff1e3e2\": container with ID starting with 7b4274f5e71fc6a82355b76c97e5554c897e19ce89d3a0c939945db10ff1e3e2 not found: ID does not exist" containerID="7b4274f5e71fc6a82355b76c97e5554c897e19ce89d3a0c939945db10ff1e3e2" Oct 30 19:34:07.635742 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.635711 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4274f5e71fc6a82355b76c97e5554c897e19ce89d3a0c939945db10ff1e3e2"} err="failed to get container status \"7b4274f5e71fc6a82355b76c97e5554c897e19ce89d3a0c939945db10ff1e3e2\": rpc error: code = NotFound desc = could not find container \"7b4274f5e71fc6a82355b76c97e5554c897e19ce89d3a0c939945db10ff1e3e2\": container with ID starting with 7b4274f5e71fc6a82355b76c97e5554c897e19ce89d3a0c939945db10ff1e3e2 not found: ID does not exist" Oct 30 19:34:07.658735 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.658668 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" podStartSLOduration=2.65864395 podStartE2EDuration="2.65864395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:07.653172737 +0000 UTC m=+1800.727628097" watchObservedRunningTime="2026-10-30 19:34:07.65864395 +0000 UTC m=+1800.733099309" Oct 30 19:34:07.679323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.679280 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istiod-ca-cert\") pod \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " Oct 30 19:34:07.679520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.679360 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-data\") pod \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " Oct 30 19:34:07.679520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.679388 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-istio-envoy\") pod \"708f310c-157e-4c22-8de1-1bdecacc2023\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " Oct 30 19:34:07.679520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.679420 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-token\" (UniqueName: \"kubernetes.io/projected/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-token\") pod \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " Oct 30 19:34:07.679520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.679469 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-workload-certs\") pod \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " Oct 30 19:34:07.679520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.679496 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-credential-socket\") pod \"708f310c-157e-4c22-8de1-1bdecacc2023\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " Oct 30 19:34:07.679767 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.679527 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/586ca704-ecaf-4d51-88b4-6ea6770bb10f-kube-api-access-g7m96\" (UniqueName: \"kubernetes.io/projected/586ca704-ecaf-4d51-88b4-6ea6770bb10f-kube-api-access-g7m96\") pod \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " Oct 30 19:34:07.679767 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.679658 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-podinfo\") pod \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " Oct 30 19:34:07.679767 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.679707 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-data" pod "586ca704-ecaf-4d51-88b4-6ea6770bb10f" (UID: "586ca704-ecaf-4d51-88b4-6ea6770bb10f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.679767 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.679732 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708f310c-157e-4c22-8de1-1bdecacc2023-istio-token\" (UniqueName: \"kubernetes.io/projected/708f310c-157e-4c22-8de1-1bdecacc2023-istio-token\") pod \"708f310c-157e-4c22-8de1-1bdecacc2023\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " Oct 30 19:34:07.680066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.679759 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istiod-ca-cert" pod "586ca704-ecaf-4d51-88b4-6ea6770bb10f" (UID: "586ca704-ecaf-4d51-88b4-6ea6770bb10f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:07.680066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.679766 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-envoy\") pod \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " Oct 30 19:34:07.680066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.679948 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-credential-socket" pod "708f310c-157e-4c22-8de1-1bdecacc2023" (UID: "708f310c-157e-4c22-8de1-1bdecacc2023"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.680066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.679863 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-workload-socket\") pod \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " Oct 30 19:34:07.680217 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.680072 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708f310c-157e-4c22-8de1-1bdecacc2023-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708f310c-157e-4c22-8de1-1bdecacc2023-istiod-ca-cert\") pod \"708f310c-157e-4c22-8de1-1bdecacc2023\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " Oct 30 19:34:07.680217 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.680141 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/708f310c-157e-4c22-8de1-1bdecacc2023-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708f310c-157e-4c22-8de1-1bdecacc2023-istio-podinfo\") pod \"708f310c-157e-4c22-8de1-1bdecacc2023\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " Oct 30 19:34:07.680296 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.680229 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708f310c-157e-4c22-8de1-1bdecacc2023-kube-api-access-hb5zk\" (UniqueName: \"kubernetes.io/projected/708f310c-157e-4c22-8de1-1bdecacc2023-kube-api-access-hb5zk\") pod \"708f310c-157e-4c22-8de1-1bdecacc2023\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " Oct 30 19:34:07.680340 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.680330 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-workload-certs\") pod \"708f310c-157e-4c22-8de1-1bdecacc2023\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " Oct 30 19:34:07.680395 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.680377 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-istio-data\") pod \"708f310c-157e-4c22-8de1-1bdecacc2023\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " Oct 30 19:34:07.680471 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.680436 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-credential-socket\") pod \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\" (UID: \"586ca704-ecaf-4d51-88b4-6ea6770bb10f\") " Oct 30 19:34:07.681480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.680643 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-workload-socket\") pod \"708f310c-157e-4c22-8de1-1bdecacc2023\" (UID: \"708f310c-157e-4c22-8de1-1bdecacc2023\") " Oct 30 19:34:07.681480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.680858 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-workload-certs" pod "586ca704-ecaf-4d51-88b4-6ea6770bb10f" (UID: "586ca704-ecaf-4d51-88b4-6ea6770bb10f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.682582 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.682553 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.682729 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.682589 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.682729 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.682605 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.682729 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.682620 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.683526 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.683243 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-workload-socket" pod "586ca704-ecaf-4d51-88b4-6ea6770bb10f" (UID: "586ca704-ecaf-4d51-88b4-6ea6770bb10f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.683636 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.683587 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-istio-data" pod "708f310c-157e-4c22-8de1-1bdecacc2023" (UID: "708f310c-157e-4c22-8de1-1bdecacc2023"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.684135 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.683869 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708f310c-157e-4c22-8de1-1bdecacc2023-istiod-ca-cert" pod "708f310c-157e-4c22-8de1-1bdecacc2023" (UID: "708f310c-157e-4c22-8de1-1bdecacc2023"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:07.684135 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.683889 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-credential-socket" pod "586ca704-ecaf-4d51-88b4-6ea6770bb10f" (UID: "586ca704-ecaf-4d51-88b4-6ea6770bb10f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.684135 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.684078 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-envoy" pod "586ca704-ecaf-4d51-88b4-6ea6770bb10f" (UID: "586ca704-ecaf-4d51-88b4-6ea6770bb10f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.684341 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.684220 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-workload-certs" pod "708f310c-157e-4c22-8de1-1bdecacc2023" (UID: "708f310c-157e-4c22-8de1-1bdecacc2023"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.684516 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.684462 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-workload-socket" pod "708f310c-157e-4c22-8de1-1bdecacc2023" (UID: "708f310c-157e-4c22-8de1-1bdecacc2023"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.685207 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.685182 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-token" pod "586ca704-ecaf-4d51-88b4-6ea6770bb10f" (UID: "586ca704-ecaf-4d51-88b4-6ea6770bb10f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:07.685782 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.685742 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708f310c-157e-4c22-8de1-1bdecacc2023-istio-token" pod "708f310c-157e-4c22-8de1-1bdecacc2023" (UID: "708f310c-157e-4c22-8de1-1bdecacc2023"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:07.685871 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.685844 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-podinfo" pod "586ca704-ecaf-4d51-88b4-6ea6770bb10f" (UID: "586ca704-ecaf-4d51-88b4-6ea6770bb10f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:34:07.687561 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.687530 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-istio-envoy" pod "708f310c-157e-4c22-8de1-1bdecacc2023" (UID: "708f310c-157e-4c22-8de1-1bdecacc2023"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.688652 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.688618 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/708f310c-157e-4c22-8de1-1bdecacc2023-istio-podinfo" pod "708f310c-157e-4c22-8de1-1bdecacc2023" (UID: "708f310c-157e-4c22-8de1-1bdecacc2023"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:34:07.689688 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.689659 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708f310c-157e-4c22-8de1-1bdecacc2023-kube-api-access-hb5zk" pod "708f310c-157e-4c22-8de1-1bdecacc2023" (UID: "708f310c-157e-4c22-8de1-1bdecacc2023"). InnerVolumeSpecName "kube-api-access-hb5zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:07.690735 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.690704 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586ca704-ecaf-4d51-88b4-6ea6770bb10f-kube-api-access-g7m96" pod "586ca704-ecaf-4d51-88b4-6ea6770bb10f" (UID: "586ca704-ecaf-4d51-88b4-6ea6770bb10f"). InnerVolumeSpecName "kube-api-access-g7m96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:07.737659 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.737220 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd"] Oct 30 19:34:07.744419 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.744356 5047 scope.go:122] "RemoveContainer" containerID="e47cd3dfe05e82d988c03f5cecafe8c7b4ff400a2c4883f34dbd29dad24639b3" Oct 30 19:34:07.744883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.744854 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-mzdxd"] Oct 30 19:34:07.780773 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.780715 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:07.792155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.784176 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.792155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.784203 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708f310c-157e-4c22-8de1-1bdecacc2023-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.792155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.784215 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708f310c-157e-4c22-8de1-1bdecacc2023-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.792155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.784228 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb5zk\" (UniqueName: \"kubernetes.io/projected/708f310c-157e-4c22-8de1-1bdecacc2023-kube-api-access-hb5zk\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.792155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.784242 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.792155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.784254 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.792155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.784264 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.792155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.784273 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.792155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.784281 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708f310c-157e-4c22-8de1-1bdecacc2023-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.792155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.784289 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.792155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.784297 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7m96\" (UniqueName: \"kubernetes.io/projected/586ca704-ecaf-4d51-88b4-6ea6770bb10f-kube-api-access-g7m96\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.792155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.784306 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.792155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.784314 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708f310c-157e-4c22-8de1-1bdecacc2023-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.792155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.784324 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586ca704-ecaf-4d51-88b4-6ea6770bb10f-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:07.792155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.785447 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:34:07.792155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.789325 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:07.794746 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.794709 5047 scope.go:122] "RemoveContainer" containerID="be8cd1ecb2db0cc524b9c5aad84833f17c7d32edd9222b650338660bf689cb4c" Oct 30 19:34:07.916161 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.914385 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-istio-data\") pod \"bccbf42c-7950-485f-a5b4-0475f827c419\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " Oct 30 19:34:07.916161 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.914473 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bccbf42c-7950-485f-a5b4-0475f827c419-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bccbf42c-7950-485f-a5b4-0475f827c419-istio-podinfo\") pod \"bccbf42c-7950-485f-a5b4-0475f827c419\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " Oct 30 19:34:07.916161 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.914583 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-istio-envoy\") pod \"bccbf42c-7950-485f-a5b4-0475f827c419\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " Oct 30 19:34:07.916161 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.914622 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-workload-certs\") pod \"bccbf42c-7950-485f-a5b4-0475f827c419\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " Oct 30 19:34:07.916161 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.914776 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bccbf42c-7950-485f-a5b4-0475f827c419-kube-api-access-2h68z\" (UniqueName: \"kubernetes.io/projected/bccbf42c-7950-485f-a5b4-0475f827c419-kube-api-access-2h68z\") pod \"bccbf42c-7950-485f-a5b4-0475f827c419\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " Oct 30 19:34:07.916161 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.914864 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bccbf42c-7950-485f-a5b4-0475f827c419-istio-token\" (UniqueName: \"kubernetes.io/projected/bccbf42c-7950-485f-a5b4-0475f827c419-istio-token\") pod \"bccbf42c-7950-485f-a5b4-0475f827c419\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " Oct 30 19:34:07.916161 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.914924 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-credential-socket\") pod \"bccbf42c-7950-485f-a5b4-0475f827c419\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " Oct 30 19:34:07.916161 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.915099 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-workload-socket\") pod \"bccbf42c-7950-485f-a5b4-0475f827c419\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " Oct 30 19:34:07.916161 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.915159 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bccbf42c-7950-485f-a5b4-0475f827c419-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bccbf42c-7950-485f-a5b4-0475f827c419-istiod-ca-cert\") pod \"bccbf42c-7950-485f-a5b4-0475f827c419\" (UID: \"bccbf42c-7950-485f-a5b4-0475f827c419\") " Oct 30 19:34:07.921725 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.919887 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-workload-certs" pod "bccbf42c-7950-485f-a5b4-0475f827c419" (UID: "bccbf42c-7950-485f-a5b4-0475f827c419"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.921725 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.921393 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-istio-data" pod "bccbf42c-7950-485f-a5b4-0475f827c419" (UID: "bccbf42c-7950-485f-a5b4-0475f827c419"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.958490 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.940174 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-workload-socket" pod "bccbf42c-7950-485f-a5b4-0475f827c419" (UID: "bccbf42c-7950-485f-a5b4-0475f827c419"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.958490 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.947155 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-credential-socket" pod "bccbf42c-7950-485f-a5b4-0475f827c419" (UID: "bccbf42c-7950-485f-a5b4-0475f827c419"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:07.958490 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.958441 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bccbf42c-7950-485f-a5b4-0475f827c419-istiod-ca-cert" pod "bccbf42c-7950-485f-a5b4-0475f827c419" (UID: "bccbf42c-7950-485f-a5b4-0475f827c419"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:08.031209 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.986658 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:08.031209 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.986698 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:08.031209 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.986713 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bccbf42c-7950-485f-a5b4-0475f827c419-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:08.031209 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.986726 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:08.031209 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.986738 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:08.031209 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:07.987631 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bccbf42c-7950-485f-a5b4-0475f827c419-istio-token" pod "bccbf42c-7950-485f-a5b4-0475f827c419" (UID: "bccbf42c-7950-485f-a5b4-0475f827c419"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:08.068951 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.068891 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-istio-envoy" pod "bccbf42c-7950-485f-a5b4-0475f827c419" (UID: "bccbf42c-7950-485f-a5b4-0475f827c419"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:08.070990 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.070946 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bccbf42c-7950-485f-a5b4-0475f827c419-kube-api-access-2h68z" pod "bccbf42c-7950-485f-a5b4-0475f827c419" (UID: "bccbf42c-7950-485f-a5b4-0475f827c419"). InnerVolumeSpecName "kube-api-access-2h68z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:08.072507 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.072241 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bccbf42c-7950-485f-a5b4-0475f827c419-istio-podinfo" pod "bccbf42c-7950-485f-a5b4-0475f827c419" (UID: "bccbf42c-7950-485f-a5b4-0475f827c419"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:34:08.091368 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.089961 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bccbf42c-7950-485f-a5b4-0475f827c419-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:08.091368 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.089993 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bccbf42c-7950-485f-a5b4-0475f827c419-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:08.091368 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.090038 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h68z\" (UniqueName: \"kubernetes.io/projected/bccbf42c-7950-485f-a5b4-0475f827c419-kube-api-access-2h68z\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:08.091368 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.090056 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bccbf42c-7950-485f-a5b4-0475f827c419-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:08.098309 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.098237 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95"] Oct 30 19:34:08.111639 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.111614 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-wqj95"] Oct 30 19:34:08.213737 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.213699 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l"] Oct 30 19:34:08.214499 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.214482 5047 cpu_manager.go:414] "containerMap: removing container" podUID="9e15d741-48fd-4962-90db-de0e6cb664f3" containerName="istio-proxy" Oct 30 19:34:08.214555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.214499 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e15d741-48fd-4962-90db-de0e6cb664f3" containerName="istio-proxy" Oct 30 19:34:08.214555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.214517 5047 cpu_manager.go:414] "containerMap: removing container" podUID="bccbf42c-7950-485f-a5b4-0475f827c419" containerName="istio-proxy" Oct 30 19:34:08.214555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.214525 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bccbf42c-7950-485f-a5b4-0475f827c419" containerName="istio-proxy" Oct 30 19:34:08.214555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.214534 5047 cpu_manager.go:414] "containerMap: removing container" podUID="9e15d741-48fd-4962-90db-de0e6cb664f3" containerName="istio-proxy" Oct 30 19:34:08.214555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.214541 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e15d741-48fd-4962-90db-de0e6cb664f3" containerName="istio-proxy" Oct 30 19:34:08.214747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.214558 5047 cpu_manager.go:414] "containerMap: removing container" podUID="586ca704-ecaf-4d51-88b4-6ea6770bb10f" containerName="istio-proxy" Oct 30 19:34:08.214747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.214563 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="586ca704-ecaf-4d51-88b4-6ea6770bb10f" containerName="istio-proxy" Oct 30 19:34:08.214747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.214582 5047 cpu_manager.go:414] "containerMap: removing container" podUID="708f310c-157e-4c22-8de1-1bdecacc2023" containerName="istio-proxy" Oct 30 19:34:08.214747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.214587 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708f310c-157e-4c22-8de1-1bdecacc2023" containerName="istio-proxy" Oct 30 19:34:08.214747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.214704 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e15d741-48fd-4962-90db-de0e6cb664f3" containerName="istio-proxy" Oct 30 19:34:08.214747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.214714 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="bccbf42c-7950-485f-a5b4-0475f827c419" containerName="istio-proxy" Oct 30 19:34:08.214747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.214720 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e15d741-48fd-4962-90db-de0e6cb664f3" containerName="istio-proxy" Oct 30 19:34:08.214747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.214729 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="586ca704-ecaf-4d51-88b4-6ea6770bb10f" containerName="istio-proxy" Oct 30 19:34:08.214747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.214737 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="708f310c-157e-4c22-8de1-1bdecacc2023" containerName="istio-proxy" Oct 30 19:34:08.218780 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.218759 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" Oct 30 19:34:08.234485 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.234453 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f4ab4a\"" Oct 30 19:34:08.235059 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.235035 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f4ab4a-dockercfg-tdg42\"" Oct 30 19:34:08.241068 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.240986 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l"] Oct 30 19:34:08.276026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.275970 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62"] Oct 30 19:34:08.358349 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.358313 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:34:08.370032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.368927 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht2c7\" (UniqueName: \"kubernetes.io/projected/6a0824a6-857e-47a0-9bff-5b0ed8ce38e9-kube-api-access-ht2c7\") pod \"payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l\" (UID: \"6a0824a6-857e-47a0-9bff-5b0ed8ce38e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" Oct 30 19:34:08.403576 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.387152 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0824a6-857e-47a0-9bff-5b0ed8ce38e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l\" (UID: \"6a0824a6-857e-47a0-9bff-5b0ed8ce38e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" Oct 30 19:34:08.405402 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.405339 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-092bc0-serving-cert\"" Oct 30 19:34:08.467294 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.460082 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x"] Oct 30 19:34:08.757024 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.745139 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b414f8-8bde-4da5-9a34-d42552256a73-openshift-service-ca\") pod \"maas-api-e2e-api-a-092bc0-85498957b8-lcs62\" (UID: \"43b414f8-8bde-4da5-9a34-d42552256a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:34:08.757024 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.745513 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43b414f8-8bde-4da5-9a34-d42552256a73-maas-api-tls\") pod \"maas-api-e2e-api-a-092bc0-85498957b8-lcs62\" (UID: \"43b414f8-8bde-4da5-9a34-d42552256a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:34:08.757537 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.757497 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b414f8-8bde-4da5-9a34-d42552256a73-metrics-certs\") pod \"maas-api-e2e-api-a-092bc0-85498957b8-lcs62\" (UID: \"43b414f8-8bde-4da5-9a34-d42552256a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:34:08.757650 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.757632 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht2c7\" (UniqueName: \"kubernetes.io/projected/6a0824a6-857e-47a0-9bff-5b0ed8ce38e9-kube-api-access-ht2c7\") pod \"payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l\" (UID: \"6a0824a6-857e-47a0-9bff-5b0ed8ce38e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" Oct 30 19:34:08.757787 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.757772 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0824a6-857e-47a0-9bff-5b0ed8ce38e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l\" (UID: \"6a0824a6-857e-47a0-9bff-5b0ed8ce38e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" Oct 30 19:34:08.757857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.757815 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqr5d\" (UniqueName: \"kubernetes.io/projected/43b414f8-8bde-4da5-9a34-d42552256a73-kube-api-access-mqr5d\") pod \"maas-api-e2e-api-a-092bc0-85498957b8-lcs62\" (UID: \"43b414f8-8bde-4da5-9a34-d42552256a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:34:08.765515 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.764451 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0824a6-857e-47a0-9bff-5b0ed8ce38e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l\" (UID: \"6a0824a6-857e-47a0-9bff-5b0ed8ce38e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" Oct 30 19:34:08.765515 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.764831 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62"] Oct 30 19:34:08.765515 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.764964 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" Oct 30 19:34:08.905067 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.884582 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht2c7\" (UniqueName: \"kubernetes.io/projected/6a0824a6-857e-47a0-9bff-5b0ed8ce38e9-kube-api-access-ht2c7\") pod \"payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l\" (UID: \"6a0824a6-857e-47a0-9bff-5b0ed8ce38e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" Oct 30 19:34:08.961071 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.961023 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqr5d\" (UniqueName: \"kubernetes.io/projected/43b414f8-8bde-4da5-9a34-d42552256a73-kube-api-access-mqr5d\") pod \"maas-api-e2e-api-a-092bc0-85498957b8-lcs62\" (UID: \"43b414f8-8bde-4da5-9a34-d42552256a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:34:08.961493 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.961468 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b414f8-8bde-4da5-9a34-d42552256a73-openshift-service-ca\") pod \"maas-api-e2e-api-a-092bc0-85498957b8-lcs62\" (UID: \"43b414f8-8bde-4da5-9a34-d42552256a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:34:08.961943 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.961920 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43b414f8-8bde-4da5-9a34-d42552256a73-maas-api-tls\") pod \"maas-api-e2e-api-a-092bc0-85498957b8-lcs62\" (UID: \"43b414f8-8bde-4da5-9a34-d42552256a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:34:08.962150 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.962133 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b414f8-8bde-4da5-9a34-d42552256a73-metrics-certs\") pod \"maas-api-e2e-api-a-092bc0-85498957b8-lcs62\" (UID: \"43b414f8-8bde-4da5-9a34-d42552256a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:34:08.973333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.973302 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b414f8-8bde-4da5-9a34-d42552256a73-openshift-service-ca\") pod \"maas-api-e2e-api-a-092bc0-85498957b8-lcs62\" (UID: \"43b414f8-8bde-4da5-9a34-d42552256a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:34:08.977543 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.976978 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" Oct 30 19:34:09.027046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.982344 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" Oct 30 19:34:09.027046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:08.999035 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqr5d\" (UniqueName: \"kubernetes.io/projected/43b414f8-8bde-4da5-9a34-d42552256a73-kube-api-access-mqr5d\") pod \"maas-api-e2e-api-a-092bc0-85498957b8-lcs62\" (UID: \"43b414f8-8bde-4da5-9a34-d42552256a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:34:09.027046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.001113 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:09.027046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.023392 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:34:09.027046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.023425 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" event={"ID":"586ca704-ecaf-4d51-88b4-6ea6770bb10f","Type":"ContainerDied","Data":"d52759bf76bc9e6ac70849c373248bb4d73cb23fffa6aeab94874534e39853f2"} Oct 30 19:34:09.027046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.023473 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg" event={"ID":"bccbf42c-7950-485f-a5b4-0475f827c419","Type":"ContainerDied","Data":"9efd9cbdc5f30109e725d1103e4b2b00f404cea8a7b80759b19ef2bb36ddc98a"} Oct 30 19:34:09.027046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.001241 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5" Oct 30 19:34:09.029034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.028923 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b414f8-8bde-4da5-9a34-d42552256a73-metrics-certs\") pod \"maas-api-e2e-api-a-092bc0-85498957b8-lcs62\" (UID: \"43b414f8-8bde-4da5-9a34-d42552256a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:34:09.043180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.043150 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43b414f8-8bde-4da5-9a34-d42552256a73-maas-api-tls\") pod \"maas-api-e2e-api-a-092bc0-85498957b8-lcs62\" (UID: \"43b414f8-8bde-4da5-9a34-d42552256a73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:34:09.072316 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.071581 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d3adfad-8cca-4e98-9812-8c17c9cfb4c1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x\" (UID: \"7d3adfad-8cca-4e98-9812-8c17c9cfb4c1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" Oct 30 19:34:09.072316 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.071983 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6c9t\" (UniqueName: \"kubernetes.io/projected/7d3adfad-8cca-4e98-9812-8c17c9cfb4c1-kube-api-access-g6c9t\") pod \"payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x\" (UID: \"7d3adfad-8cca-4e98-9812-8c17c9cfb4c1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" Oct 30 19:34:09.072316 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.072134 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:34:09.073249 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.073219 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708f310c-157e-4c22-8de1-1bdecacc2023" path="/var/lib/kubelet/pods/708f310c-157e-4c22-8de1-1bdecacc2023/volumes" Oct 30 19:34:09.073978 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.073956 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e15d741-48fd-4962-90db-de0e6cb664f3" path="/var/lib/kubelet/pods/9e15d741-48fd-4962-90db-de0e6cb664f3/volumes" Oct 30 19:34:09.074835 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.074813 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x"] Oct 30 19:34:09.153439 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.153378 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg"] Oct 30 19:34:09.160368 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.160331 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-2brlg"] Oct 30 19:34:09.179017 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.178570 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d3adfad-8cca-4e98-9812-8c17c9cfb4c1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x\" (UID: \"7d3adfad-8cca-4e98-9812-8c17c9cfb4c1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" Oct 30 19:34:09.179017 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.178666 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6c9t\" (UniqueName: \"kubernetes.io/projected/7d3adfad-8cca-4e98-9812-8c17c9cfb4c1-kube-api-access-g6c9t\") pod \"payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x\" (UID: \"7d3adfad-8cca-4e98-9812-8c17c9cfb4c1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" Oct 30 19:34:09.179560 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.179497 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d3adfad-8cca-4e98-9812-8c17c9cfb4c1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x\" (UID: \"7d3adfad-8cca-4e98-9812-8c17c9cfb4c1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" Oct 30 19:34:09.189243 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:34:09.189204 5047 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbccbf42c_7950_485f_a5b4_0475f827c419.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586ca704_ecaf_4d51_88b4_6ea6770bb10f.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:34:09.204862 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.204825 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6c9t\" (UniqueName: \"kubernetes.io/projected/7d3adfad-8cca-4e98-9812-8c17c9cfb4c1-kube-api-access-g6c9t\") pod \"payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x\" (UID: \"7d3adfad-8cca-4e98-9812-8c17c9cfb4c1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" Oct 30 19:34:09.270543 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.270448 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5"] Oct 30 19:34:09.293822 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.277938 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9efc72-openshift-default-d4ddf7dcc-sxjk5"] Oct 30 19:34:09.352096 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.347945 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" Oct 30 19:34:09.558747 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:34:09.558715 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0824a6_857e_47a0_9bff_5b0ed8ce38e9.slice/crio-da682d3050e5f4579803c676304e7bd9534898e3a0eeffd4e2354cc726274124 WatchSource:0}: Error finding container da682d3050e5f4579803c676304e7bd9534898e3a0eeffd4e2354cc726274124: Status 404 returned error can't find the container with id da682d3050e5f4579803c676304e7bd9534898e3a0eeffd4e2354cc726274124 Oct 30 19:34:09.560789 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.560763 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l"] Oct 30 19:34:09.608740 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:34:09.608703 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:34:09.630172 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.630134 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62"] Oct 30 19:34:09.631168 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:34:09.631135 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b414f8_8bde_4da5_9a34_d42552256a73.slice/crio-7b61af7dbfbd947c11921ec6b30d4d0d6edcece39aca480b6ca9d380f5edd2b2 WatchSource:0}: Error finding container 7b61af7dbfbd947c11921ec6b30d4d0d6edcece39aca480b6ca9d380f5edd2b2: Status 404 returned error can't find the container with id 7b61af7dbfbd947c11921ec6b30d4d0d6edcece39aca480b6ca9d380f5edd2b2 Oct 30 19:34:09.632825 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.632804 5047 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:34:09.844118 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.843280 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt"] Oct 30 19:34:09.877155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.877112 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x"] Oct 30 19:34:09.877721 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.877690 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:09.946354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.927368 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:09.946354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.927531 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:09.946354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.927586 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:09.946354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.927656 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b913225f-a9cf-4af0-af6f-076a9c0ec683-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:09.946354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.927683 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:09.946354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.927741 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:09.946354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.927773 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:09.946354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.927858 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:09.946354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.927899 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97rkd\" (UniqueName: \"kubernetes.io/projected/b913225f-a9cf-4af0-af6f-076a9c0ec683-kube-api-access-97rkd\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:09.962206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.955965 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b"] Oct 30 19:34:09.974172 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.970288 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt"] Oct 30 19:34:09.974172 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.973785 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:09.974172 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:09.974113 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b"] Oct 30 19:34:10.001352 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.001312 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" event={"ID":"43b414f8-8bde-4da5-9a34-d42552256a73","Type":"ContainerStarted","Data":"7b61af7dbfbd947c11921ec6b30d4d0d6edcece39aca480b6ca9d380f5edd2b2"} Oct 30 19:34:10.003385 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.003354 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" event={"ID":"6a0824a6-857e-47a0-9bff-5b0ed8ce38e9","Type":"ContainerStarted","Data":"f49f3c9ec78854bc03d585c8a1255199dbe3b8e9e51e99213afe8e9ba2df949c"} Oct 30 19:34:10.003687 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.003592 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" Oct 30 19:34:10.003815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.003800 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" event={"ID":"6a0824a6-857e-47a0-9bff-5b0ed8ce38e9","Type":"ContainerStarted","Data":"da682d3050e5f4579803c676304e7bd9534898e3a0eeffd4e2354cc726274124"} Oct 30 19:34:10.006571 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.005529 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" event={"ID":"7d3adfad-8cca-4e98-9812-8c17c9cfb4c1","Type":"ContainerStarted","Data":"baae557c64746b2fed340fe42e5bf1a361ca89352ce65c828d00d87aad1aaf93"} Oct 30 19:34:10.029398 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.029307 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.029766 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.029746 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.029895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.029876 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.030087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.030071 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.030193 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.030178 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.030298 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.030285 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.030453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.030440 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b913225f-a9cf-4af0-af6f-076a9c0ec683-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.030539 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.030527 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.030675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.030660 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/581eaa37-1f68-4519-a69a-3f4fcee15057-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.030787 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.030772 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.030919 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.030880 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.031098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.031082 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.033790 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.033755 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.033918 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.033803 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97rkd\" (UniqueName: \"kubernetes.io/projected/b913225f-a9cf-4af0-af6f-076a9c0ec683-kube-api-access-97rkd\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.033978 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.033953 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.034168 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.034148 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfrb4\" (UniqueName: \"kubernetes.io/projected/581eaa37-1f68-4519-a69a-3f4fcee15057-kube-api-access-nfrb4\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.034229 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.034188 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.034229 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.032979 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b913225f-a9cf-4af0-af6f-076a9c0ec683-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.034229 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.032043 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.034393 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.034233 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.034393 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.032924 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.034393 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.033510 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.034588 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.034566 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.113067 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.108046 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97rkd\" (UniqueName: \"kubernetes.io/projected/b913225f-a9cf-4af0-af6f-076a9c0ec683-kube-api-access-97rkd\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.116202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.116116 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.119724 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.119693 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.122468 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.122435 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.136346 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.136312 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.136490 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.136467 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.136633 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.136616 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfrb4\" (UniqueName: \"kubernetes.io/projected/581eaa37-1f68-4519-a69a-3f4fcee15057-kube-api-access-nfrb4\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.136701 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.136661 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.136774 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.136753 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.136904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.136885 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.136981 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.136966 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.137098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.137048 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.137158 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.137112 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.137158 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.137140 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/581eaa37-1f68-4519-a69a-3f4fcee15057-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.138344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.137728 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.138344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.137979 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.138344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.138308 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.138737 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.138712 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/581eaa37-1f68-4519-a69a-3f4fcee15057-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.140132 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.140111 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.140532 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.140511 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.156969 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.156936 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.157187 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.157114 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfrb4\" (UniqueName: \"kubernetes.io/projected/581eaa37-1f68-4519-a69a-3f4fcee15057-kube-api-access-nfrb4\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.305340 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.305279 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" podStartSLOduration=2.305258693 podStartE2EDuration="2.305258693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:10.109073637 +0000 UTC m=+1803.183529001" watchObservedRunningTime="2026-10-30 19:34:10.305258693 +0000 UTC m=+1803.379714054" Oct 30 19:34:10.352105 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.352000 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:10.460031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.432894 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:10.678032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.677909 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs"] Oct 30 19:34:10.925428 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.925393 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:10.953487 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.953414 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-287ea2-openshift-default-dockercfg-7kw4v\"" Oct 30 19:34:10.962215 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:10.962178 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs"] Oct 30 19:34:11.079779 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.079606 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="586ca704-ecaf-4d51-88b4-6ea6770bb10f" path="/var/lib/kubelet/pods/586ca704-ecaf-4d51-88b4-6ea6770bb10f/volumes" Oct 30 19:34:11.081074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.081048 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bccbf42c-7950-485f-a5b4-0475f827c419" path="/var/lib/kubelet/pods/bccbf42c-7950-485f-a5b4-0475f827c419/volumes" Oct 30 19:34:11.112904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.109909 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a252f556-2d81-4164-8efa-65037946f382-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.112904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.110109 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.112904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.110193 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a252f556-2d81-4164-8efa-65037946f382-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.112904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.110366 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.112904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.110400 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.112904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.110557 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a252f556-2d81-4164-8efa-65037946f382-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.112904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.110632 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tqng\" (UniqueName: \"kubernetes.io/projected/a252f556-2d81-4164-8efa-65037946f382-kube-api-access-8tqng\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.112904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.110791 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.112904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.110930 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.138026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.137338 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" event={"ID":"43b414f8-8bde-4da5-9a34-d42552256a73","Type":"ContainerStarted","Data":"3aab7dd12a27c782a379595f3d383982214f38e899081c3c67d295bfd86fa55a"} Oct 30 19:34:11.138250 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.138118 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:34:11.139986 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.139962 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" event={"ID":"7d3adfad-8cca-4e98-9812-8c17c9cfb4c1","Type":"ContainerStarted","Data":"c7e2592193c532f309801930c9dea7af8868a96f3f98a21263c029c1b579251a"} Oct 30 19:34:11.140230 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.140214 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" Oct 30 19:34:11.184690 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.184633 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" podStartSLOduration=2.809501612 podStartE2EDuration="3.184614969s" totalImagesPullingTime="375.113357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:11.181738327 +0000 UTC m=+1804.256193686" watchObservedRunningTime="2026-10-30 19:34:11.184614969 +0000 UTC m=+1804.259070374" Oct 30 19:34:11.213189 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.213085 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a252f556-2d81-4164-8efa-65037946f382-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.213189 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.213181 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.213623 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.213587 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.214228 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.214129 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.214228 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.214139 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.214411 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.214298 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a252f556-2d81-4164-8efa-65037946f382-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.214411 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.214369 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tqng\" (UniqueName: \"kubernetes.io/projected/a252f556-2d81-4164-8efa-65037946f382-kube-api-access-8tqng\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.214514 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.214492 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.214896 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.214854 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.216889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.215235 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a252f556-2d81-4164-8efa-65037946f382-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.216889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.215400 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.216889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.215563 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.216889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.216501 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a252f556-2d81-4164-8efa-65037946f382-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.216889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.216600 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.218481 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.218443 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.221398 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.221339 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" podStartSLOduration=3.22132151 podStartE2EDuration="3.22132151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:11.220936713 +0000 UTC m=+1804.295392072" watchObservedRunningTime="2026-10-30 19:34:11.22132151 +0000 UTC m=+1804.295776870" Oct 30 19:34:11.222356 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.222312 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a252f556-2d81-4164-8efa-65037946f382-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.238406 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.238337 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a252f556-2d81-4164-8efa-65037946f382-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.238701 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.238646 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tqng\" (UniqueName: \"kubernetes.io/projected/a252f556-2d81-4164-8efa-65037946f382-kube-api-access-8tqng\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.277824 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.277783 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:11.478736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.474815 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt"] Oct 30 19:34:11.533862 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.533832 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b"] Oct 30 19:34:11.534144 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:34:11.534115 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581eaa37_1f68_4519_a69a_3f4fcee15057.slice/crio-78045f39d87766faf2c787da5d1616d77c6d20fb11cc663ed3168a04b6d09ec8 WatchSource:0}: Error finding container 78045f39d87766faf2c787da5d1616d77c6d20fb11cc663ed3168a04b6d09ec8: Status 404 returned error can't find the container with id 78045f39d87766faf2c787da5d1616d77c6d20fb11cc663ed3168a04b6d09ec8 Oct 30 19:34:11.534220 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.534183 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:11.534290 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.534271 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:11.534346 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.534326 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:11.541799 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.537524 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:11.541799 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.537621 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:11.541799 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.537691 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:11.604324 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:34:11.604279 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda252f556_2d81_4164_8efa_65037946f382.slice/crio-9ddbe4e70debfc1f7faadc2e775c6617a49da7b10ae2d67277c0890cbde8fa69 WatchSource:0}: Error finding container 9ddbe4e70debfc1f7faadc2e775c6617a49da7b10ae2d67277c0890cbde8fa69: Status 404 returned error can't find the container with id 9ddbe4e70debfc1f7faadc2e775c6617a49da7b10ae2d67277c0890cbde8fa69 Oct 30 19:34:11.605070 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.605043 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs"] Oct 30 19:34:11.608104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.608068 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:11.608205 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.608143 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:11.608205 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:11.608175 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:12.156758 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:12.156715 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" event={"ID":"a252f556-2d81-4164-8efa-65037946f382","Type":"ContainerStarted","Data":"a1348b944313cd6a43577eb3d495ef73a1c9e16ed41a301461d8fa11773d4559"} Oct 30 19:34:12.157554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:12.156786 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" event={"ID":"a252f556-2d81-4164-8efa-65037946f382","Type":"ContainerStarted","Data":"9ddbe4e70debfc1f7faadc2e775c6617a49da7b10ae2d67277c0890cbde8fa69"} Oct 30 19:34:12.159569 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:12.159516 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" event={"ID":"b913225f-a9cf-4af0-af6f-076a9c0ec683","Type":"ContainerStarted","Data":"8b40dc18b077766fa4a849a56062cf6dc7ea6135eb0969a71834480c484d80c5"} Oct 30 19:34:12.159712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:12.159629 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" event={"ID":"b913225f-a9cf-4af0-af6f-076a9c0ec683","Type":"ContainerStarted","Data":"dda261ab9bf4cf378471d3239f65594bb09eb99d4089e3c7f46859e8808f983e"} Oct 30 19:34:12.161593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:12.161559 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" event={"ID":"581eaa37-1f68-4519-a69a-3f4fcee15057","Type":"ContainerStarted","Data":"d7186411c8b531586afb6bef871723146d1f7042d10103c8f08c1bcac37e42fe"} Oct 30 19:34:12.161737 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:12.161618 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" event={"ID":"581eaa37-1f68-4519-a69a-3f4fcee15057","Type":"ContainerStarted","Data":"78045f39d87766faf2c787da5d1616d77c6d20fb11cc663ed3168a04b6d09ec8"} Oct 30 19:34:12.207127 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:12.207050 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" podStartSLOduration=2.2070258 podStartE2EDuration="2.2070258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:12.198767306 +0000 UTC m=+1805.273222678" watchObservedRunningTime="2026-10-30 19:34:12.2070258 +0000 UTC m=+1805.281481154" Oct 30 19:34:12.289138 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:12.289101 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:12.293855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:12.293793 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" podStartSLOduration=3.293773785 podStartE2EDuration="3.293773785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:12.284860012 +0000 UTC m=+1805.359315372" watchObservedRunningTime="2026-10-30 19:34:12.293773785 +0000 UTC m=+1805.368229144" Oct 30 19:34:12.297081 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:12.297042 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:12.368170 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:12.368069 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:12.389060 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:12.384212 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" podStartSLOduration=3.384189937 podStartE2EDuration="3.384189937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:12.330517161 +0000 UTC m=+1805.404972533" watchObservedRunningTime="2026-10-30 19:34:12.384189937 +0000 UTC m=+1805.458645298" Oct 30 19:34:12.398255 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:12.392130 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:34:12.398255 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:12.392200 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" podUID="b913225f-a9cf-4af0-af6f-076a9c0ec683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 19:34:12.445940 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:12.445759 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:12.447234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:12.447199 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 19:34:12.447408 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:12.447356 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" podUID="581eaa37-1f68-4519-a69a-3f4fcee15057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 19:34:13.349030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:13.336727 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:13.367409 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:13.350168 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:34:13.367409 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:13.357782 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:34:13.367409 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:13.357875 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" podUID="b913225f-a9cf-4af0-af6f-076a9c0ec683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 19:34:13.457031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:13.440794 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 19:34:13.457031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:13.440865 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" podUID="581eaa37-1f68-4519-a69a-3f4fcee15057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 19:34:14.358154 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:14.358113 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:34:14.358776 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:14.358740 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" podUID="b913225f-a9cf-4af0-af6f-076a9c0ec683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 19:34:14.455053 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:14.454525 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 19:34:14.455053 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:14.454595 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" podUID="581eaa37-1f68-4519-a69a-3f4fcee15057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 19:34:15.355116 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:15.354540 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:34:15.355116 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:15.354608 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" podUID="b913225f-a9cf-4af0-af6f-076a9c0ec683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 19:34:15.439495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:15.439449 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 19:34:15.440131 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:15.440098 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" podUID="581eaa37-1f68-4519-a69a-3f4fcee15057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 19:34:16.360094 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:16.358180 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:34:16.360094 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:16.358242 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" podUID="b913225f-a9cf-4af0-af6f-076a9c0ec683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 19:34:16.461246 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:16.461204 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 19:34:16.461926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:16.461866 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" podUID="581eaa37-1f68-4519-a69a-3f4fcee15057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 19:34:17.188033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:17.186877 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:34:17.359492 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:17.359443 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:34:17.379347 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:17.379302 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" podUID="b913225f-a9cf-4af0-af6f-076a9c0ec683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 19:34:17.459281 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:17.459183 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 19:34:17.464136 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:17.464087 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" podUID="581eaa37-1f68-4519-a69a-3f4fcee15057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 19:34:18.421203 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:18.421174 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:18.422096 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:18.422069 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:18.426961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:18.426429 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 19:34:18.448324 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:18.448287 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:18.450386 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:18.450363 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:18.453329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:18.453303 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:34:21.144318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:21.144286 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" Oct 30 19:34:22.164601 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:22.164570 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" Oct 30 19:34:27.235955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:27.235920 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6"] Oct 30 19:34:27.236453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:27.235983 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p"] Oct 30 19:34:27.236453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:27.236092 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44"] Oct 30 19:34:27.236453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:27.236134 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r"] Oct 30 19:34:27.236453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:27.236318 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" podUID="58531637-258f-49e4-996d-088591db67e6" containerName="istio-proxy" containerID="cri-o://b94f63b780a15ae9610be20f8fa3d4ddedd5c4d7d2f27a2efdfaf2701cd020e8" gracePeriod=30 Oct 30 19:34:27.236624 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:27.236564 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" podUID="97681db1-a8e8-4ad9-a97f-b914a8c55c02" containerName="istio-proxy" containerID="cri-o://3aaeb2d989d2a421c1b446bfee87458ea7d9bafa62e15fdfd746a61aa5e82443" gracePeriod=30 Oct 30 19:34:27.236744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:27.236720 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" podUID="d2ceec35-f806-4418-aebd-f515b2423b4d" containerName="istio-proxy" containerID="cri-o://5e7ecde9e7baefadb4ce1b829d0a2b381539d74ab19b67bab530c3f3d1eec9d4" gracePeriod=30 Oct 30 19:34:27.237515 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:27.237484 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" podUID="709e2b00-e43d-425d-855f-8b498883c049" containerName="istio-proxy" containerID="cri-o://1d5e042c5872cac499956b583b10fad0b3cbf7ccdd639a38a914e0deaa7ea46a" gracePeriod=30 Oct 30 19:34:32.626608 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.626583 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:34:32.630789 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.630763 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:34:32.684558 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.684523 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709e2b00-e43d-425d-855f-8b498883c049-istio-token\" (UniqueName: \"kubernetes.io/projected/709e2b00-e43d-425d-855f-8b498883c049-istio-token\") pod \"709e2b00-e43d-425d-855f-8b498883c049\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " Oct 30 19:34:32.684737 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.684607 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-workload-certs\") pod \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " Oct 30 19:34:32.684737 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.684651 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-envoy\") pod \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " Oct 30 19:34:32.684737 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.684705 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-workload-certs\") pod \"709e2b00-e43d-425d-855f-8b498883c049\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " Oct 30 19:34:32.684902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.684741 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/709e2b00-e43d-425d-855f-8b498883c049-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709e2b00-e43d-425d-855f-8b498883c049-istio-podinfo\") pod \"709e2b00-e43d-425d-855f-8b498883c049\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " Oct 30 19:34:32.684902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.684801 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/709e2b00-e43d-425d-855f-8b498883c049-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709e2b00-e43d-425d-855f-8b498883c049-istiod-ca-cert\") pod \"709e2b00-e43d-425d-855f-8b498883c049\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " Oct 30 19:34:32.685024 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.684917 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-istio-envoy\") pod \"709e2b00-e43d-425d-855f-8b498883c049\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " Oct 30 19:34:32.685024 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.684956 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-workload-certs" pod "97681db1-a8e8-4ad9-a97f-b914a8c55c02" (UID: "97681db1-a8e8-4ad9-a97f-b914a8c55c02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.685024 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.685001 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-podinfo\") pod \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " Oct 30 19:34:32.685198 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.685080 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istiod-ca-cert\") pod \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " Oct 30 19:34:32.685198 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.685142 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-credential-socket\") pod \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " Oct 30 19:34:32.685198 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.685175 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709e2b00-e43d-425d-855f-8b498883c049-kube-api-access-kwdnw\" (UniqueName: \"kubernetes.io/projected/709e2b00-e43d-425d-855f-8b498883c049-kube-api-access-kwdnw\") pod \"709e2b00-e43d-425d-855f-8b498883c049\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " Oct 30 19:34:32.685357 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.685217 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-workload-socket\") pod \"709e2b00-e43d-425d-855f-8b498883c049\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " Oct 30 19:34:32.685357 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.685281 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-token\" (UniqueName: \"kubernetes.io/projected/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-token\") pod \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " Oct 30 19:34:32.685357 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.685314 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709e2b00-e43d-425d-855f-8b498883c049-istiod-ca-cert" pod "709e2b00-e43d-425d-855f-8b498883c049" (UID: "709e2b00-e43d-425d-855f-8b498883c049"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:32.685357 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.685331 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-workload-certs" pod "709e2b00-e43d-425d-855f-8b498883c049" (UID: "709e2b00-e43d-425d-855f-8b498883c049"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.685571 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.685509 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-data\") pod \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " Oct 30 19:34:32.685571 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.685549 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-workload-socket\") pod \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " Oct 30 19:34:32.685674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.685584 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-credential-socket\") pod \"709e2b00-e43d-425d-855f-8b498883c049\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " Oct 30 19:34:32.685674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.685612 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-istio-data\") pod \"709e2b00-e43d-425d-855f-8b498883c049\" (UID: \"709e2b00-e43d-425d-855f-8b498883c049\") " Oct 30 19:34:32.685870 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.685679 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97681db1-a8e8-4ad9-a97f-b914a8c55c02-kube-api-access-lfzgz\" (UniqueName: \"kubernetes.io/projected/97681db1-a8e8-4ad9-a97f-b914a8c55c02-kube-api-access-lfzgz\") pod \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\" (UID: \"97681db1-a8e8-4ad9-a97f-b914a8c55c02\") " Oct 30 19:34:32.685870 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.685707 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istiod-ca-cert" pod "97681db1-a8e8-4ad9-a97f-b914a8c55c02" (UID: "97681db1-a8e8-4ad9-a97f-b914a8c55c02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:32.686714 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.686686 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/709e2b00-e43d-425d-855f-8b498883c049-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.686844 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.686718 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.686844 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.686732 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.686844 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.686746 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.687918 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.687891 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-workload-socket" pod "709e2b00-e43d-425d-855f-8b498883c049" (UID: "709e2b00-e43d-425d-855f-8b498883c049"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.688159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.688128 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709e2b00-e43d-425d-855f-8b498883c049-istio-token" pod "709e2b00-e43d-425d-855f-8b498883c049" (UID: "709e2b00-e43d-425d-855f-8b498883c049"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:32.688842 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.688816 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-workload-socket" pod "97681db1-a8e8-4ad9-a97f-b914a8c55c02" (UID: "97681db1-a8e8-4ad9-a97f-b914a8c55c02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.689062 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.688995 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-data" pod "97681db1-a8e8-4ad9-a97f-b914a8c55c02" (UID: "97681db1-a8e8-4ad9-a97f-b914a8c55c02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.689237 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.689218 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-credential-socket" pod "709e2b00-e43d-425d-855f-8b498883c049" (UID: "709e2b00-e43d-425d-855f-8b498883c049"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.689403 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.689386 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-istio-data" pod "709e2b00-e43d-425d-855f-8b498883c049" (UID: "709e2b00-e43d-425d-855f-8b498883c049"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.689582 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.689565 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-credential-socket" pod "97681db1-a8e8-4ad9-a97f-b914a8c55c02" (UID: "97681db1-a8e8-4ad9-a97f-b914a8c55c02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.689879 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.689853 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-istio-envoy" pod "709e2b00-e43d-425d-855f-8b498883c049" (UID: "709e2b00-e43d-425d-855f-8b498883c049"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.690515 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.690482 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-token" pod "97681db1-a8e8-4ad9-a97f-b914a8c55c02" (UID: "97681db1-a8e8-4ad9-a97f-b914a8c55c02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:32.690620 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.690513 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-envoy" pod "97681db1-a8e8-4ad9-a97f-b914a8c55c02" (UID: "97681db1-a8e8-4ad9-a97f-b914a8c55c02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.690620 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.690560 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/709e2b00-e43d-425d-855f-8b498883c049-istio-podinfo" pod "709e2b00-e43d-425d-855f-8b498883c049" (UID: "709e2b00-e43d-425d-855f-8b498883c049"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:34:32.690735 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.690717 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709e2b00-e43d-425d-855f-8b498883c049-kube-api-access-kwdnw" pod "709e2b00-e43d-425d-855f-8b498883c049" (UID: "709e2b00-e43d-425d-855f-8b498883c049"). InnerVolumeSpecName "kube-api-access-kwdnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:32.690909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.690888 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97681db1-a8e8-4ad9-a97f-b914a8c55c02-kube-api-access-lfzgz" pod "97681db1-a8e8-4ad9-a97f-b914a8c55c02" (UID: "97681db1-a8e8-4ad9-a97f-b914a8c55c02"). InnerVolumeSpecName "kube-api-access-lfzgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:32.691341 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.691314 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-podinfo" pod "97681db1-a8e8-4ad9-a97f-b914a8c55c02" (UID: "97681db1-a8e8-4ad9-a97f-b914a8c55c02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:34:32.712644 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.712602 5047 generic.go:356] "Generic (PLEG): container finished" podID="97681db1-a8e8-4ad9-a97f-b914a8c55c02" containerID="3aaeb2d989d2a421c1b446bfee87458ea7d9bafa62e15fdfd746a61aa5e82443" exitCode=0 Oct 30 19:34:32.712836 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.712725 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" event={"ID":"97681db1-a8e8-4ad9-a97f-b914a8c55c02","Type":"ContainerDied","Data":"3aaeb2d989d2a421c1b446bfee87458ea7d9bafa62e15fdfd746a61aa5e82443"} Oct 30 19:34:32.712836 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.712784 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" event={"ID":"97681db1-a8e8-4ad9-a97f-b914a8c55c02","Type":"ContainerDied","Data":"1ce29c3dfbd69c570c71c18496232db1cc5f5fcc08e48fc0284d0b98ca98daad"} Oct 30 19:34:32.712836 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.712816 5047 scope.go:122] "RemoveContainer" containerID="3aaeb2d989d2a421c1b446bfee87458ea7d9bafa62e15fdfd746a61aa5e82443" Oct 30 19:34:32.713069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.712983 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6" Oct 30 19:34:32.718839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.717108 5047 generic.go:356] "Generic (PLEG): container finished" podID="d2ceec35-f806-4418-aebd-f515b2423b4d" containerID="5e7ecde9e7baefadb4ce1b829d0a2b381539d74ab19b67bab530c3f3d1eec9d4" exitCode=0 Oct 30 19:34:32.718839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.717194 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" event={"ID":"d2ceec35-f806-4418-aebd-f515b2423b4d","Type":"ContainerDied","Data":"5e7ecde9e7baefadb4ce1b829d0a2b381539d74ab19b67bab530c3f3d1eec9d4"} Oct 30 19:34:32.719478 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.719245 5047 generic.go:356] "Generic (PLEG): container finished" podID="709e2b00-e43d-425d-855f-8b498883c049" containerID="1d5e042c5872cac499956b583b10fad0b3cbf7ccdd639a38a914e0deaa7ea46a" exitCode=0 Oct 30 19:34:32.720575 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.719652 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" event={"ID":"709e2b00-e43d-425d-855f-8b498883c049","Type":"ContainerDied","Data":"1d5e042c5872cac499956b583b10fad0b3cbf7ccdd639a38a914e0deaa7ea46a"} Oct 30 19:34:32.720575 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.719715 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" event={"ID":"709e2b00-e43d-425d-855f-8b498883c049","Type":"ContainerDied","Data":"fc60f1bf8382f62ef8a4358c84b3d265e6181f99d80ba34f100dee6a8c463e81"} Oct 30 19:34:32.720575 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.719801 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44" Oct 30 19:34:32.726130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.724777 5047 generic.go:356] "Generic (PLEG): container finished" podID="58531637-258f-49e4-996d-088591db67e6" containerID="b94f63b780a15ae9610be20f8fa3d4ddedd5c4d7d2f27a2efdfaf2701cd020e8" exitCode=0 Oct 30 19:34:32.726130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.726073 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" event={"ID":"58531637-258f-49e4-996d-088591db67e6","Type":"ContainerDied","Data":"b94f63b780a15ae9610be20f8fa3d4ddedd5c4d7d2f27a2efdfaf2701cd020e8"} Oct 30 19:34:32.742788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.742761 5047 scope.go:122] "RemoveContainer" containerID="3aaeb2d989d2a421c1b446bfee87458ea7d9bafa62e15fdfd746a61aa5e82443" Oct 30 19:34:32.743454 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:34:32.743423 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aaeb2d989d2a421c1b446bfee87458ea7d9bafa62e15fdfd746a61aa5e82443\": container with ID starting with 3aaeb2d989d2a421c1b446bfee87458ea7d9bafa62e15fdfd746a61aa5e82443 not found: ID does not exist" containerID="3aaeb2d989d2a421c1b446bfee87458ea7d9bafa62e15fdfd746a61aa5e82443" Oct 30 19:34:32.743712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.743465 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aaeb2d989d2a421c1b446bfee87458ea7d9bafa62e15fdfd746a61aa5e82443"} err="failed to get container status \"3aaeb2d989d2a421c1b446bfee87458ea7d9bafa62e15fdfd746a61aa5e82443\": rpc error: code = NotFound desc = could not find container \"3aaeb2d989d2a421c1b446bfee87458ea7d9bafa62e15fdfd746a61aa5e82443\": container with ID starting with 3aaeb2d989d2a421c1b446bfee87458ea7d9bafa62e15fdfd746a61aa5e82443 not found: ID does not exist" Oct 30 19:34:32.743712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.743491 5047 scope.go:122] "RemoveContainer" containerID="1d5e042c5872cac499956b583b10fad0b3cbf7ccdd639a38a914e0deaa7ea46a" Oct 30 19:34:32.759845 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.759813 5047 scope.go:122] "RemoveContainer" containerID="1d5e042c5872cac499956b583b10fad0b3cbf7ccdd639a38a914e0deaa7ea46a" Oct 30 19:34:32.760327 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:34:32.760295 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d5e042c5872cac499956b583b10fad0b3cbf7ccdd639a38a914e0deaa7ea46a\": container with ID starting with 1d5e042c5872cac499956b583b10fad0b3cbf7ccdd639a38a914e0deaa7ea46a not found: ID does not exist" containerID="1d5e042c5872cac499956b583b10fad0b3cbf7ccdd639a38a914e0deaa7ea46a" Oct 30 19:34:32.760431 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.760355 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d5e042c5872cac499956b583b10fad0b3cbf7ccdd639a38a914e0deaa7ea46a"} err="failed to get container status \"1d5e042c5872cac499956b583b10fad0b3cbf7ccdd639a38a914e0deaa7ea46a\": rpc error: code = NotFound desc = could not find container \"1d5e042c5872cac499956b583b10fad0b3cbf7ccdd639a38a914e0deaa7ea46a\": container with ID starting with 1d5e042c5872cac499956b583b10fad0b3cbf7ccdd639a38a914e0deaa7ea46a not found: ID does not exist" Oct 30 19:34:32.775938 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.775899 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6"] Oct 30 19:34:32.780432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.780405 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-nvks6"] Oct 30 19:34:32.789168 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.789140 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfzgz\" (UniqueName: \"kubernetes.io/projected/97681db1-a8e8-4ad9-a97f-b914a8c55c02-kube-api-access-lfzgz\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.789297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.789173 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709e2b00-e43d-425d-855f-8b498883c049-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.789297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.789189 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.789297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.789202 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/709e2b00-e43d-425d-855f-8b498883c049-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.789297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.789214 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.789297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.789228 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.789297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.789241 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.789297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.789258 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwdnw\" (UniqueName: \"kubernetes.io/projected/709e2b00-e43d-425d-855f-8b498883c049-kube-api-access-kwdnw\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.789297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.789271 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.789297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.789286 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.789297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.789298 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.789622 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.789311 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97681db1-a8e8-4ad9-a97f-b914a8c55c02-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.789622 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.789324 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.789622 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.789337 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/709e2b00-e43d-425d-855f-8b498883c049-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:32.829429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.829399 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44"] Oct 30 19:34:32.834219 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:32.834192 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-wnc44"] Oct 30 19:34:33.003775 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.003729 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:34:33.032082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.032050 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.178891 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-istio-envoy\") pod \"58531637-258f-49e4-996d-088591db67e6\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.178958 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-credential-socket\") pod \"58531637-258f-49e4-996d-088591db67e6\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.179075 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-workload-socket\") pod \"58531637-258f-49e4-996d-088591db67e6\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.179155 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-token\" (UniqueName: \"kubernetes.io/projected/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-token\") pod \"d2ceec35-f806-4418-aebd-f515b2423b4d\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.179192 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2ceec35-f806-4418-aebd-f515b2423b4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2ceec35-f806-4418-aebd-f515b2423b4d-istiod-ca-cert\") pod \"d2ceec35-f806-4418-aebd-f515b2423b4d\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.179255 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-workload-certs\") pod \"d2ceec35-f806-4418-aebd-f515b2423b4d\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.179318 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58531637-258f-49e4-996d-088591db67e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58531637-258f-49e4-996d-088591db67e6-istiod-ca-cert\") pod \"58531637-258f-49e4-996d-088591db67e6\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.179457 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58531637-258f-49e4-996d-088591db67e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58531637-258f-49e4-996d-088591db67e6-istio-podinfo\") pod \"58531637-258f-49e4-996d-088591db67e6\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.179521 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58531637-258f-49e4-996d-088591db67e6-kube-api-access-5r6c4\" (UniqueName: \"kubernetes.io/projected/58531637-258f-49e4-996d-088591db67e6-kube-api-access-5r6c4\") pod \"58531637-258f-49e4-996d-088591db67e6\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.179572 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-credential-socket\") pod \"d2ceec35-f806-4418-aebd-f515b2423b4d\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.179609 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-data\") pod \"d2ceec35-f806-4418-aebd-f515b2423b4d\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.179663 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-envoy\") pod \"d2ceec35-f806-4418-aebd-f515b2423b4d\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.179696 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-podinfo\") pod \"d2ceec35-f806-4418-aebd-f515b2423b4d\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.179753 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-workload-certs\") pod \"58531637-258f-49e4-996d-088591db67e6\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.179834 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-istio-data\") pod \"58531637-258f-49e4-996d-088591db67e6\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.179888 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ceec35-f806-4418-aebd-f515b2423b4d-kube-api-access-q6p77\" (UniqueName: \"kubernetes.io/projected/d2ceec35-f806-4418-aebd-f515b2423b4d-kube-api-access-q6p77\") pod \"d2ceec35-f806-4418-aebd-f515b2423b4d\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.179949 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58531637-258f-49e4-996d-088591db67e6-istio-token\" (UniqueName: \"kubernetes.io/projected/58531637-258f-49e4-996d-088591db67e6-istio-token\") pod \"58531637-258f-49e4-996d-088591db67e6\" (UID: \"58531637-258f-49e4-996d-088591db67e6\") " Oct 30 19:34:33.180083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.179986 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-workload-socket\") pod \"d2ceec35-f806-4418-aebd-f515b2423b4d\" (UID: \"d2ceec35-f806-4418-aebd-f515b2423b4d\") " Oct 30 19:34:33.208067 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.189709 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58531637-258f-49e4-996d-088591db67e6-istiod-ca-cert" pod "58531637-258f-49e4-996d-088591db67e6" (UID: "58531637-258f-49e4-996d-088591db67e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:33.208067 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.200380 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-credential-socket" pod "58531637-258f-49e4-996d-088591db67e6" (UID: "58531637-258f-49e4-996d-088591db67e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:33.208067 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.206069 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-data" pod "d2ceec35-f806-4418-aebd-f515b2423b4d" (UID: "d2ceec35-f806-4418-aebd-f515b2423b4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:33.208067 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.207041 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-istio-envoy" pod "58531637-258f-49e4-996d-088591db67e6" (UID: "58531637-258f-49e4-996d-088591db67e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:33.220418 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.220377 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-workload-certs" pod "58531637-258f-49e4-996d-088591db67e6" (UID: "58531637-258f-49e4-996d-088591db67e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:33.220913 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.220874 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-istio-data" pod "58531637-258f-49e4-996d-088591db67e6" (UID: "58531637-258f-49e4-996d-088591db67e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:33.222727 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.222653 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ceec35-f806-4418-aebd-f515b2423b4d-istiod-ca-cert" pod "d2ceec35-f806-4418-aebd-f515b2423b4d" (UID: "d2ceec35-f806-4418-aebd-f515b2423b4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:33.222908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.222882 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-workload-certs" pod "d2ceec35-f806-4418-aebd-f515b2423b4d" (UID: "d2ceec35-f806-4418-aebd-f515b2423b4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:33.231348 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.229460 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-credential-socket" pod "d2ceec35-f806-4418-aebd-f515b2423b4d" (UID: "d2ceec35-f806-4418-aebd-f515b2423b4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:33.245815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.233681 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-workload-socket" pod "58531637-258f-49e4-996d-088591db67e6" (UID: "58531637-258f-49e4-996d-088591db67e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:33.245815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.234657 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-workload-socket" pod "d2ceec35-f806-4418-aebd-f515b2423b4d" (UID: "d2ceec35-f806-4418-aebd-f515b2423b4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:33.245815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.235401 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="709e2b00-e43d-425d-855f-8b498883c049" path="/var/lib/kubelet/pods/709e2b00-e43d-425d-855f-8b498883c049/volumes" Oct 30 19:34:33.245815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.236538 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97681db1-a8e8-4ad9-a97f-b914a8c55c02" path="/var/lib/kubelet/pods/97681db1-a8e8-4ad9-a97f-b914a8c55c02/volumes" Oct 30 19:34:33.268421 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.268355 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58531637-258f-49e4-996d-088591db67e6-istio-token" pod "58531637-258f-49e4-996d-088591db67e6" (UID: "58531637-258f-49e4-996d-088591db67e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:33.270620 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.270410 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58531637-258f-49e4-996d-088591db67e6-istio-podinfo" pod "58531637-258f-49e4-996d-088591db67e6" (UID: "58531637-258f-49e4-996d-088591db67e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:34:33.270620 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.270454 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-token" pod "d2ceec35-f806-4418-aebd-f515b2423b4d" (UID: "d2ceec35-f806-4418-aebd-f515b2423b4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:33.270620 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.270540 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-envoy" pod "d2ceec35-f806-4418-aebd-f515b2423b4d" (UID: "d2ceec35-f806-4418-aebd-f515b2423b4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:33.270868 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.270659 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-podinfo" pod "d2ceec35-f806-4418-aebd-f515b2423b4d" (UID: "d2ceec35-f806-4418-aebd-f515b2423b4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:34:33.271909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.271886 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58531637-258f-49e4-996d-088591db67e6-kube-api-access-5r6c4" pod "58531637-258f-49e4-996d-088591db67e6" (UID: "58531637-258f-49e4-996d-088591db67e6"). InnerVolumeSpecName "kube-api-access-5r6c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:33.273483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.273455 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ceec35-f806-4418-aebd-f515b2423b4d-kube-api-access-q6p77" pod "d2ceec35-f806-4418-aebd-f515b2423b4d" (UID: "d2ceec35-f806-4418-aebd-f515b2423b4d"). InnerVolumeSpecName "kube-api-access-q6p77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:33.283673 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283643 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.283811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283676 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.283811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283689 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6p77\" (UniqueName: \"kubernetes.io/projected/d2ceec35-f806-4418-aebd-f515b2423b4d-kube-api-access-q6p77\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.283811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283704 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58531637-258f-49e4-996d-088591db67e6-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.283811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283716 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.283811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283728 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.283811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283740 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.283811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283752 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58531637-258f-49e4-996d-088591db67e6-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.283811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283765 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.283811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283777 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2ceec35-f806-4418-aebd-f515b2423b4d-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.283811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283789 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.283811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283800 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58531637-258f-49e4-996d-088591db67e6-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.283811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283812 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58531637-258f-49e4-996d-088591db67e6-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.284328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283824 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r6c4\" (UniqueName: \"kubernetes.io/projected/58531637-258f-49e4-996d-088591db67e6-kube-api-access-5r6c4\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.284328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283837 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.284328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283851 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.284328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283864 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.284328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.283877 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2ceec35-f806-4418-aebd-f515b2423b4d-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:33.735158 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.735118 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" event={"ID":"d2ceec35-f806-4418-aebd-f515b2423b4d","Type":"ContainerDied","Data":"82617253e57025889110b7319a99da9a5288c20f2b0ab69eece691be429af7fd"} Oct 30 19:34:33.735158 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.735137 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p" Oct 30 19:34:33.735688 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.735182 5047 scope.go:122] "RemoveContainer" containerID="5e7ecde9e7baefadb4ce1b829d0a2b381539d74ab19b67bab530c3f3d1eec9d4" Oct 30 19:34:33.737924 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.737902 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" event={"ID":"58531637-258f-49e4-996d-088591db67e6","Type":"ContainerDied","Data":"a9edf2f3a59a4430f66e68865cf51c4b32de426a7afa781ed7b0d21ae15f4beb"} Oct 30 19:34:33.738100 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.737946 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r" Oct 30 19:34:33.750734 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.750710 5047 scope.go:122] "RemoveContainer" containerID="b94f63b780a15ae9610be20f8fa3d4ddedd5c4d7d2f27a2efdfaf2701cd020e8" Oct 30 19:34:33.800765 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.800731 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r"] Oct 30 19:34:33.812517 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.812477 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-44d6r"] Oct 30 19:34:33.844917 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.844882 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p"] Oct 30 19:34:33.848944 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:33.848921 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b97379-openshift-default-79f7b9f759-bpq8p"] Oct 30 19:34:35.051318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:35.051282 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58531637-258f-49e4-996d-088591db67e6" path="/var/lib/kubelet/pods/58531637-258f-49e4-996d-088591db67e6/volumes" Oct 30 19:34:35.052074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:35.052052 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2ceec35-f806-4418-aebd-f515b2423b4d" path="/var/lib/kubelet/pods/d2ceec35-f806-4418-aebd-f515b2423b4d/volumes" Oct 30 19:34:37.956634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:37.956596 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6"] Oct 30 19:34:37.957649 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:37.957624 5047 cpu_manager.go:414] "containerMap: removing container" podUID="709e2b00-e43d-425d-855f-8b498883c049" containerName="istio-proxy" Oct 30 19:34:37.957649 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:37.957647 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709e2b00-e43d-425d-855f-8b498883c049" containerName="istio-proxy" Oct 30 19:34:37.957831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:37.957683 5047 cpu_manager.go:414] "containerMap: removing container" podUID="d2ceec35-f806-4418-aebd-f515b2423b4d" containerName="istio-proxy" Oct 30 19:34:37.957831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:37.957689 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ceec35-f806-4418-aebd-f515b2423b4d" containerName="istio-proxy" Oct 30 19:34:37.957831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:37.957711 5047 cpu_manager.go:414] "containerMap: removing container" podUID="97681db1-a8e8-4ad9-a97f-b914a8c55c02" containerName="istio-proxy" Oct 30 19:34:37.957831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:37.957716 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97681db1-a8e8-4ad9-a97f-b914a8c55c02" containerName="istio-proxy" Oct 30 19:34:37.957831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:37.957723 5047 cpu_manager.go:414] "containerMap: removing container" podUID="58531637-258f-49e4-996d-088591db67e6" containerName="istio-proxy" Oct 30 19:34:37.957831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:37.957728 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58531637-258f-49e4-996d-088591db67e6" containerName="istio-proxy" Oct 30 19:34:37.958130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:37.957851 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ceec35-f806-4418-aebd-f515b2423b4d" containerName="istio-proxy" Oct 30 19:34:37.958130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:37.957863 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="97681db1-a8e8-4ad9-a97f-b914a8c55c02" containerName="istio-proxy" Oct 30 19:34:37.958130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:37.957874 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="58531637-258f-49e4-996d-088591db67e6" containerName="istio-proxy" Oct 30 19:34:37.958130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:37.957890 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="709e2b00-e43d-425d-855f-8b498883c049" containerName="istio-proxy" Oct 30 19:34:37.962493 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:37.962466 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:34:37.991893 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:37.985988 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f4ab4a-serving-cert\"" Oct 30 19:34:38.022765 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:38.022722 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6"] Oct 30 19:34:38.076051 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:38.075995 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d691a1-d91c-4985-9866-3a96f94b2e67-maas-api-tls\") pod \"maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6\" (UID: \"d4d691a1-d91c-4985-9866-3a96f94b2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:34:38.076241 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:38.076219 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d691a1-d91c-4985-9866-3a96f94b2e67-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6\" (UID: \"d4d691a1-d91c-4985-9866-3a96f94b2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:34:38.076395 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:38.076380 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jwbs\" (UniqueName: \"kubernetes.io/projected/d4d691a1-d91c-4985-9866-3a96f94b2e67-kube-api-access-4jwbs\") pod \"maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6\" (UID: \"d4d691a1-d91c-4985-9866-3a96f94b2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:34:38.076429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:38.076404 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d691a1-d91c-4985-9866-3a96f94b2e67-metrics-certs\") pod \"maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6\" (UID: \"d4d691a1-d91c-4985-9866-3a96f94b2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:34:38.177700 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:38.177658 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d691a1-d91c-4985-9866-3a96f94b2e67-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6\" (UID: \"d4d691a1-d91c-4985-9866-3a96f94b2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:34:38.177916 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:38.177872 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jwbs\" (UniqueName: \"kubernetes.io/projected/d4d691a1-d91c-4985-9866-3a96f94b2e67-kube-api-access-4jwbs\") pod \"maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6\" (UID: \"d4d691a1-d91c-4985-9866-3a96f94b2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:34:38.177916 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:38.177909 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d691a1-d91c-4985-9866-3a96f94b2e67-metrics-certs\") pod \"maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6\" (UID: \"d4d691a1-d91c-4985-9866-3a96f94b2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:34:38.178134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:38.178109 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d691a1-d91c-4985-9866-3a96f94b2e67-maas-api-tls\") pod \"maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6\" (UID: \"d4d691a1-d91c-4985-9866-3a96f94b2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:34:38.178553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:38.178532 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d691a1-d91c-4985-9866-3a96f94b2e67-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6\" (UID: \"d4d691a1-d91c-4985-9866-3a96f94b2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:34:38.180536 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:38.180508 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d691a1-d91c-4985-9866-3a96f94b2e67-metrics-certs\") pod \"maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6\" (UID: \"d4d691a1-d91c-4985-9866-3a96f94b2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:34:38.180634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:38.180553 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d691a1-d91c-4985-9866-3a96f94b2e67-maas-api-tls\") pod \"maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6\" (UID: \"d4d691a1-d91c-4985-9866-3a96f94b2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:34:38.248095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:38.245685 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jwbs\" (UniqueName: \"kubernetes.io/projected/d4d691a1-d91c-4985-9866-3a96f94b2e67-kube-api-access-4jwbs\") pod \"maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6\" (UID: \"d4d691a1-d91c-4985-9866-3a96f94b2e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:34:38.284736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:38.284698 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:34:38.691074 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:34:38.691045 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d691a1_d91c_4985_9866_3a96f94b2e67.slice/crio-3d929941246015b32f00811fc315456ce9ee02145bbdc6583ad6aa42a6a7b168 WatchSource:0}: Error finding container 3d929941246015b32f00811fc315456ce9ee02145bbdc6583ad6aa42a6a7b168: Status 404 returned error can't find the container with id 3d929941246015b32f00811fc315456ce9ee02145bbdc6583ad6aa42a6a7b168 Oct 30 19:34:38.693049 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:38.693022 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6"] Oct 30 19:34:38.865718 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:38.864170 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" event={"ID":"d4d691a1-d91c-4985-9866-3a96f94b2e67","Type":"ContainerStarted","Data":"3d929941246015b32f00811fc315456ce9ee02145bbdc6583ad6aa42a6a7b168"} Oct 30 19:34:39.697261 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:34:39.696839 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:34:39.875063 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:39.874980 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" event={"ID":"d4d691a1-d91c-4985-9866-3a96f94b2e67","Type":"ContainerStarted","Data":"c2a1ae658f4e2776fee1c36b0225e171dfd2805f5bafe0364069599638439406"} Oct 30 19:34:39.875250 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:39.875094 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:34:39.927401 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:39.927333 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" podStartSLOduration=2.516832576 podStartE2EDuration="2.927318082s" totalImagesPullingTime="410.485506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:39.918083009 +0000 UTC m=+1832.992538368" watchObservedRunningTime="2026-10-30 19:34:39.927318082 +0000 UTC m=+1833.001773440" Oct 30 19:34:42.094542 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:42.094512 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq"] Oct 30 19:34:42.095183 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:42.095153 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" podUID="5c7b1347-f416-42a3-a055-0b866e894c64" containerName="istio-proxy" containerID="cri-o://acc4b7dbe508305dfd7caf239d9f7f91cbe2fd196d9cbe8263a0f8f34331fe7b" gracePeriod=30 Oct 30 19:34:42.101202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:42.101166 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv"] Oct 30 19:34:42.101414 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:42.101390 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" podUID="04defd95-0e95-4ee6-9d47-d6b2488a2f62" containerName="istio-proxy" containerID="cri-o://7e8adb1e3556cce603e59051efbb04508134441b93977ceaf6cb443730cbc7fd" gracePeriod=30 Oct 30 19:34:42.104882 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:42.104790 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d"] Oct 30 19:34:42.105200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:42.105087 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" podUID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" containerName="istio-proxy" containerID="cri-o://30226ee388745ce9ca41c8ccde0c1d49eaf56bc89387a7df0abbf1aa3a6ce660" gracePeriod=30 Oct 30 19:34:42.111105 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:42.110037 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w"] Oct 30 19:34:42.111105 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:42.110381 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" podUID="0d95b5e1-e759-430f-b0fb-461451ee6e32" containerName="istio-proxy" containerID="cri-o://395101f7331fcfba80d06b4c1c99bcc05b0f3ff05bfaec8129452b83ccce20fc" gracePeriod=30 Oct 30 19:34:45.889412 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:45.889378 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:34:46.116236 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.116186 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq"] Oct 30 19:34:46.122645 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.122357 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.142890 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.142819 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b625eb-openshift-default-dockercfg-7hdjs\"" Oct 30 19:34:46.149307 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.149277 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq"] Oct 30 19:34:46.206164 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.206084 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-podinfo\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.206379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.206223 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-workload-socket\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.206379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.206358 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj2pd\" (UniqueName: \"kubernetes.io/projected/c10e3c6c-bbf1-4606-990e-bc2898aff682-kube-api-access-vj2pd\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.206503 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.206389 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-data\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.206503 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.206449 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-workload-certs\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.206624 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.206547 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-envoy\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.206624 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.206566 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-token\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.206705 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.206635 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-credential-socket\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.206850 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.206829 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10e3c6c-bbf1-4606-990e-bc2898aff682-istiod-ca-cert\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.313028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.309370 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj2pd\" (UniqueName: \"kubernetes.io/projected/c10e3c6c-bbf1-4606-990e-bc2898aff682-kube-api-access-vj2pd\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.313028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.309432 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-data\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.313028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.309533 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-workload-certs\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.313028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.309627 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-envoy\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.313028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.309652 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-token\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.313028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.309749 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-credential-socket\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.313028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.309794 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10e3c6c-bbf1-4606-990e-bc2898aff682-istiod-ca-cert\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.313028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.309905 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-podinfo\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.313028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.309995 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-workload-socket\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.313028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.310516 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-workload-socket\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.313028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.311516 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-data\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.336037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.313803 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-credential-socket\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.372122 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.349921 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10e3c6c-bbf1-4606-990e-bc2898aff682-istiod-ca-cert\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.454313 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.445478 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-workload-certs\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.454313 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.453719 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-podinfo\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.516461 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.516386 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj2pd\" (UniqueName: \"kubernetes.io/projected/c10e3c6c-bbf1-4606-990e-bc2898aff682-kube-api-access-vj2pd\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.556642 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.556609 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-envoy\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.556915 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.556678 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-token\") pod \"e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:46.755071 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:46.754948 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:47.126486 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.126423 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq"] Oct 30 19:34:47.158507 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:34:47.158446 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10e3c6c_bbf1_4606_990e_bc2898aff682.slice/crio-6d4ebc3631b8915edde2d1b34fd3c0a264076521ba7be2b5051d162398cfc732 WatchSource:0}: Error finding container 6d4ebc3631b8915edde2d1b34fd3c0a264076521ba7be2b5051d162398cfc732: Status 404 returned error can't find the container with id 6d4ebc3631b8915edde2d1b34fd3c0a264076521ba7be2b5051d162398cfc732 Oct 30 19:34:47.216798 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.216756 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:47.216915 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.216859 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:47.216915 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.216904 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:47.386415 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.386383 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:34:47.386692 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.386628 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" podUID="0d95b5e1-e759-430f-b0fb-461451ee6e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:34:47.532593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.529481 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:34:47.532593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.530806 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:34:47.592742 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.592715 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:34:47.682117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.681487 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-credential-socket\") pod \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " Oct 30 19:34:47.682117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.681543 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-istio-envoy\") pod \"5c7b1347-f416-42a3-a055-0b866e894c64\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " Oct 30 19:34:47.682117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.681604 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-workload-socket\") pod \"5c7b1347-f416-42a3-a055-0b866e894c64\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " Oct 30 19:34:47.682117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.681650 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istiod-ca-cert\") pod \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " Oct 30 19:34:47.682117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.681716 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7b1347-f416-42a3-a055-0b866e894c64-istio-token\" (UniqueName: \"kubernetes.io/projected/5c7b1347-f416-42a3-a055-0b866e894c64-istio-token\") pod \"5c7b1347-f416-42a3-a055-0b866e894c64\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " Oct 30 19:34:47.682117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.681753 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-workload-certs\") pod \"5c7b1347-f416-42a3-a055-0b866e894c64\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " Oct 30 19:34:47.682117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.681779 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-workload-certs\") pod \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " Oct 30 19:34:47.682117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.681834 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-data\") pod \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " Oct 30 19:34:47.682117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.681895 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04defd95-0e95-4ee6-9d47-d6b2488a2f62-kube-api-access-v2tc6\" (UniqueName: \"kubernetes.io/projected/04defd95-0e95-4ee6-9d47-d6b2488a2f62-kube-api-access-v2tc6\") pod \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " Oct 30 19:34:47.682117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.681943 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-podinfo\") pod \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " Oct 30 19:34:47.682117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.682014 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7b1347-f416-42a3-a055-0b866e894c64-kube-api-access-66fg7\" (UniqueName: \"kubernetes.io/projected/5c7b1347-f416-42a3-a055-0b866e894c64-kube-api-access-66fg7\") pod \"5c7b1347-f416-42a3-a055-0b866e894c64\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " Oct 30 19:34:47.684668 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.683338 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-envoy\") pod \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " Oct 30 19:34:47.684668 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.683392 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-istio-data\") pod \"5c7b1347-f416-42a3-a055-0b866e894c64\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " Oct 30 19:34:47.684668 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.683420 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c7b1347-f416-42a3-a055-0b866e894c64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7b1347-f416-42a3-a055-0b866e894c64-istiod-ca-cert\") pod \"5c7b1347-f416-42a3-a055-0b866e894c64\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " Oct 30 19:34:47.684668 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.683448 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c7b1347-f416-42a3-a055-0b866e894c64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7b1347-f416-42a3-a055-0b866e894c64-istio-podinfo\") pod \"5c7b1347-f416-42a3-a055-0b866e894c64\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " Oct 30 19:34:47.684668 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.683505 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-workload-socket\") pod \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " Oct 30 19:34:47.684668 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.683538 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-credential-socket\") pod \"5c7b1347-f416-42a3-a055-0b866e894c64\" (UID: \"5c7b1347-f416-42a3-a055-0b866e894c64\") " Oct 30 19:34:47.684668 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.683654 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-token\" (UniqueName: \"kubernetes.io/projected/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-token\") pod \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\" (UID: \"04defd95-0e95-4ee6-9d47-d6b2488a2f62\") " Oct 30 19:34:47.687842 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.686187 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-istio-data" pod "5c7b1347-f416-42a3-a055-0b866e894c64" (UID: "5c7b1347-f416-42a3-a055-0b866e894c64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:47.687842 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.686404 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-workload-socket" pod "5c7b1347-f416-42a3-a055-0b866e894c64" (UID: "5c7b1347-f416-42a3-a055-0b866e894c64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:47.687842 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.686636 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-workload-certs" pod "5c7b1347-f416-42a3-a055-0b866e894c64" (UID: "5c7b1347-f416-42a3-a055-0b866e894c64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:47.687842 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.686888 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-workload-certs" pod "04defd95-0e95-4ee6-9d47-d6b2488a2f62" (UID: "04defd95-0e95-4ee6-9d47-d6b2488a2f62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:47.687842 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.687137 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-credential-socket" pod "04defd95-0e95-4ee6-9d47-d6b2488a2f62" (UID: "04defd95-0e95-4ee6-9d47-d6b2488a2f62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:47.687842 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.687278 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-data" pod "04defd95-0e95-4ee6-9d47-d6b2488a2f62" (UID: "04defd95-0e95-4ee6-9d47-d6b2488a2f62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:47.687842 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.687401 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-workload-socket" pod "04defd95-0e95-4ee6-9d47-d6b2488a2f62" (UID: "04defd95-0e95-4ee6-9d47-d6b2488a2f62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:47.687842 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.687647 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-credential-socket" pod "5c7b1347-f416-42a3-a055-0b866e894c64" (UID: "5c7b1347-f416-42a3-a055-0b866e894c64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:47.687842 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.687656 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7b1347-f416-42a3-a055-0b866e894c64-istiod-ca-cert" pod "5c7b1347-f416-42a3-a055-0b866e894c64" (UID: "5c7b1347-f416-42a3-a055-0b866e894c64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:47.687842 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.687745 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-token" pod "04defd95-0e95-4ee6-9d47-d6b2488a2f62" (UID: "04defd95-0e95-4ee6-9d47-d6b2488a2f62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:47.688523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.688506 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istiod-ca-cert" pod "04defd95-0e95-4ee6-9d47-d6b2488a2f62" (UID: "04defd95-0e95-4ee6-9d47-d6b2488a2f62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:47.689928 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.689905 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-envoy" pod "04defd95-0e95-4ee6-9d47-d6b2488a2f62" (UID: "04defd95-0e95-4ee6-9d47-d6b2488a2f62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:47.690105 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.689944 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04defd95-0e95-4ee6-9d47-d6b2488a2f62-kube-api-access-v2tc6" pod "04defd95-0e95-4ee6-9d47-d6b2488a2f62" (UID: "04defd95-0e95-4ee6-9d47-d6b2488a2f62"). InnerVolumeSpecName "kube-api-access-v2tc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:47.690271 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.690234 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c7b1347-f416-42a3-a055-0b866e894c64-istio-podinfo" pod "5c7b1347-f416-42a3-a055-0b866e894c64" (UID: "5c7b1347-f416-42a3-a055-0b866e894c64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:34:47.690543 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.690505 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7b1347-f416-42a3-a055-0b866e894c64-kube-api-access-66fg7" pod "5c7b1347-f416-42a3-a055-0b866e894c64" (UID: "5c7b1347-f416-42a3-a055-0b866e894c64"). InnerVolumeSpecName "kube-api-access-66fg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:47.691328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.691303 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7b1347-f416-42a3-a055-0b866e894c64-istio-token" pod "5c7b1347-f416-42a3-a055-0b866e894c64" (UID: "5c7b1347-f416-42a3-a055-0b866e894c64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:47.691723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.691697 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-podinfo" pod "04defd95-0e95-4ee6-9d47-d6b2488a2f62" (UID: "04defd95-0e95-4ee6-9d47-d6b2488a2f62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:34:47.691869 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.691844 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-istio-envoy" pod "5c7b1347-f416-42a3-a055-0b866e894c64" (UID: "5c7b1347-f416-42a3-a055-0b866e894c64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:47.822377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.801953 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d95b5e1-e759-430f-b0fb-461451ee6e32-kube-api-access-2tndr\" (UniqueName: \"kubernetes.io/projected/0d95b5e1-e759-430f-b0fb-461451ee6e32-kube-api-access-2tndr\") pod \"0d95b5e1-e759-430f-b0fb-461451ee6e32\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " Oct 30 19:34:47.822377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.802147 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-credential-socket\") pod \"0d95b5e1-e759-430f-b0fb-461451ee6e32\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " Oct 30 19:34:47.822377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.802241 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-token\" (UniqueName: \"kubernetes.io/projected/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-token\") pod \"0d95b5e1-e759-430f-b0fb-461451ee6e32\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " Oct 30 19:34:47.822377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.802276 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-data\") pod \"0d95b5e1-e759-430f-b0fb-461451ee6e32\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " Oct 30 19:34:47.822377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.802376 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-envoy\") pod \"0d95b5e1-e759-430f-b0fb-461451ee6e32\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " Oct 30 19:34:47.854469 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.848785 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-credential-socket" pod "0d95b5e1-e759-430f-b0fb-461451ee6e32" (UID: "0d95b5e1-e759-430f-b0fb-461451ee6e32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.877151 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-podinfo\") pod \"0d95b5e1-e759-430f-b0fb-461451ee6e32\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.877306 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d95b5e1-e759-430f-b0fb-461451ee6e32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d95b5e1-e759-430f-b0fb-461451ee6e32-istiod-ca-cert\") pod \"0d95b5e1-e759-430f-b0fb-461451ee6e32\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.877389 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-workload-certs\") pod \"0d95b5e1-e759-430f-b0fb-461451ee6e32\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.877416 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-workload-socket\") pod \"0d95b5e1-e759-430f-b0fb-461451ee6e32\" (UID: \"0d95b5e1-e759-430f-b0fb-461451ee6e32\") " Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878430 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66fg7\" (UniqueName: \"kubernetes.io/projected/5c7b1347-f416-42a3-a055-0b866e894c64-kube-api-access-66fg7\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878458 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878473 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878487 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7b1347-f416-42a3-a055-0b866e894c64-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878500 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7b1347-f416-42a3-a055-0b866e894c64-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878513 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878526 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878538 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878554 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878567 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878579 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878593 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878608 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878620 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7b1347-f416-42a3-a055-0b866e894c64-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878634 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7b1347-f416-42a3-a055-0b866e894c64-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878649 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878665 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878678 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2tc6\" (UniqueName: \"kubernetes.io/projected/04defd95-0e95-4ee6-9d47-d6b2488a2f62-kube-api-access-v2tc6\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.880259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.878693 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04defd95-0e95-4ee6-9d47-d6b2488a2f62-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.881928 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.880528 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-workload-socket" pod "0d95b5e1-e759-430f-b0fb-461451ee6e32" (UID: "0d95b5e1-e759-430f-b0fb-461451ee6e32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:47.881928 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.881189 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-data" pod "0d95b5e1-e759-430f-b0fb-461451ee6e32" (UID: "0d95b5e1-e759-430f-b0fb-461451ee6e32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:47.881928 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.881558 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d95b5e1-e759-430f-b0fb-461451ee6e32-istiod-ca-cert" pod "0d95b5e1-e759-430f-b0fb-461451ee6e32" (UID: "0d95b5e1-e759-430f-b0fb-461451ee6e32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:47.884747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.884721 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-podinfo" pod "0d95b5e1-e759-430f-b0fb-461451ee6e32" (UID: "0d95b5e1-e759-430f-b0fb-461451ee6e32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:34:47.885478 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.885436 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-envoy" pod "0d95b5e1-e759-430f-b0fb-461451ee6e32" (UID: "0d95b5e1-e759-430f-b0fb-461451ee6e32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:47.898673 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.889124 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-workload-certs" pod "0d95b5e1-e759-430f-b0fb-461451ee6e32" (UID: "0d95b5e1-e759-430f-b0fb-461451ee6e32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:47.898673 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.890927 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d95b5e1-e759-430f-b0fb-461451ee6e32-kube-api-access-2tndr" pod "0d95b5e1-e759-430f-b0fb-461451ee6e32" (UID: "0d95b5e1-e759-430f-b0fb-461451ee6e32"). InnerVolumeSpecName "kube-api-access-2tndr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:47.947791 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.934200 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-token" pod "0d95b5e1-e759-430f-b0fb-461451ee6e32" (UID: "0d95b5e1-e759-430f-b0fb-461451ee6e32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:47.974724 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.974214 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" event={"ID":"c10e3c6c-bbf1-4606-990e-bc2898aff682","Type":"ContainerStarted","Data":"29a21cb49fc2a3a1f5364edbc0955a65f06c05ab088daede9233aefc759f059f"} Oct 30 19:34:47.974724 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.974281 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" event={"ID":"c10e3c6c-bbf1-4606-990e-bc2898aff682","Type":"ContainerStarted","Data":"6d4ebc3631b8915edde2d1b34fd3c0a264076521ba7be2b5051d162398cfc732"} Oct 30 19:34:47.977182 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.976733 5047 generic.go:356] "Generic (PLEG): container finished" podID="0d95b5e1-e759-430f-b0fb-461451ee6e32" containerID="395101f7331fcfba80d06b4c1c99bcc05b0f3ff05bfaec8129452b83ccce20fc" exitCode=0 Oct 30 19:34:47.977182 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.976970 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" event={"ID":"0d95b5e1-e759-430f-b0fb-461451ee6e32","Type":"ContainerDied","Data":"395101f7331fcfba80d06b4c1c99bcc05b0f3ff05bfaec8129452b83ccce20fc"} Oct 30 19:34:47.977182 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.977032 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" event={"ID":"0d95b5e1-e759-430f-b0fb-461451ee6e32","Type":"ContainerDied","Data":"ceda3082d97135895e9c13e0218aa3491bf75b0176124476a44e3aaebba5a3e8"} Oct 30 19:34:47.977182 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.977057 5047 scope.go:122] "RemoveContainer" containerID="395101f7331fcfba80d06b4c1c99bcc05b0f3ff05bfaec8129452b83ccce20fc" Oct 30 19:34:47.977691 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.977286 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w" Oct 30 19:34:47.981334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.980756 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.981334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.980790 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.981334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.980805 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tndr\" (UniqueName: \"kubernetes.io/projected/0d95b5e1-e759-430f-b0fb-461451ee6e32-kube-api-access-2tndr\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.981334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.980823 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.981334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.980837 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.981334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.980850 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.981334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.980863 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d95b5e1-e759-430f-b0fb-461451ee6e32-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.981334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.980875 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d95b5e1-e759-430f-b0fb-461451ee6e32-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:47.991097 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.991047 5047 generic.go:356] "Generic (PLEG): container finished" podID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" containerID="30226ee388745ce9ca41c8ccde0c1d49eaf56bc89387a7df0abbf1aa3a6ce660" exitCode=0 Oct 30 19:34:47.991240 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.991188 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" event={"ID":"10f19076-8c2f-4026-8c4b-ef3062bbe51f","Type":"ContainerDied","Data":"30226ee388745ce9ca41c8ccde0c1d49eaf56bc89387a7df0abbf1aa3a6ce660"} Oct 30 19:34:47.996180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.995269 5047 generic.go:356] "Generic (PLEG): container finished" podID="5c7b1347-f416-42a3-a055-0b866e894c64" containerID="acc4b7dbe508305dfd7caf239d9f7f91cbe2fd196d9cbe8263a0f8f34331fe7b" exitCode=0 Oct 30 19:34:47.996180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.995337 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" event={"ID":"5c7b1347-f416-42a3-a055-0b866e894c64","Type":"ContainerDied","Data":"acc4b7dbe508305dfd7caf239d9f7f91cbe2fd196d9cbe8263a0f8f34331fe7b"} Oct 30 19:34:47.996180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.995370 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" event={"ID":"5c7b1347-f416-42a3-a055-0b866e894c64","Type":"ContainerDied","Data":"fdc6872fa52a9a30b3194d6fec0347d91321533150fff2ac0a384029f7a11989"} Oct 30 19:34:47.996180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.995536 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" Oct 30 19:34:47.999424 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.999252 5047 generic.go:356] "Generic (PLEG): container finished" podID="04defd95-0e95-4ee6-9d47-d6b2488a2f62" containerID="7e8adb1e3556cce603e59051efbb04508134441b93977ceaf6cb443730cbc7fd" exitCode=0 Oct 30 19:34:47.999424 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.999321 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" Oct 30 19:34:47.999424 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.999340 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" event={"ID":"04defd95-0e95-4ee6-9d47-d6b2488a2f62","Type":"ContainerDied","Data":"7e8adb1e3556cce603e59051efbb04508134441b93977ceaf6cb443730cbc7fd"} Oct 30 19:34:47.999424 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:47.999389 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv" event={"ID":"04defd95-0e95-4ee6-9d47-d6b2488a2f62","Type":"ContainerDied","Data":"1696e0932b28c1e787b8fa5c9b9bc2fe9df318f744d142e9846f0db708524f57"} Oct 30 19:34:48.000257 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.000238 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:34:48.003392 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.003369 5047 scope.go:122] "RemoveContainer" containerID="395101f7331fcfba80d06b4c1c99bcc05b0f3ff05bfaec8129452b83ccce20fc" Oct 30 19:34:48.003887 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:34:48.003866 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"395101f7331fcfba80d06b4c1c99bcc05b0f3ff05bfaec8129452b83ccce20fc\": container with ID starting with 395101f7331fcfba80d06b4c1c99bcc05b0f3ff05bfaec8129452b83ccce20fc not found: ID does not exist" containerID="395101f7331fcfba80d06b4c1c99bcc05b0f3ff05bfaec8129452b83ccce20fc" Oct 30 19:34:48.003986 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.003892 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395101f7331fcfba80d06b4c1c99bcc05b0f3ff05bfaec8129452b83ccce20fc"} err="failed to get container status \"395101f7331fcfba80d06b4c1c99bcc05b0f3ff05bfaec8129452b83ccce20fc\": rpc error: code = NotFound desc = could not find container \"395101f7331fcfba80d06b4c1c99bcc05b0f3ff05bfaec8129452b83ccce20fc\": container with ID starting with 395101f7331fcfba80d06b4c1c99bcc05b0f3ff05bfaec8129452b83ccce20fc not found: ID does not exist" Oct 30 19:34:48.003986 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.003910 5047 scope.go:122] "RemoveContainer" containerID="b9155f826cdcab0b00b8b1c444246c9bfcfacb4e1b98fe5eb6da89f216f69de5" Oct 30 19:34:48.020484 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.020425 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" podStartSLOduration=2.020408278 podStartE2EDuration="2.020408278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:48.015258228 +0000 UTC m=+1841.089713586" watchObservedRunningTime="2026-10-30 19:34:48.020408278 +0000 UTC m=+1841.094863637" Oct 30 19:34:48.021022 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.020983 5047 scope.go:122] "RemoveContainer" containerID="acc4b7dbe508305dfd7caf239d9f7f91cbe2fd196d9cbe8263a0f8f34331fe7b" Oct 30 19:34:48.032374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.032353 5047 scope.go:122] "RemoveContainer" containerID="acc4b7dbe508305dfd7caf239d9f7f91cbe2fd196d9cbe8263a0f8f34331fe7b" Oct 30 19:34:48.032693 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:34:48.032666 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc4b7dbe508305dfd7caf239d9f7f91cbe2fd196d9cbe8263a0f8f34331fe7b\": container with ID starting with acc4b7dbe508305dfd7caf239d9f7f91cbe2fd196d9cbe8263a0f8f34331fe7b not found: ID does not exist" containerID="acc4b7dbe508305dfd7caf239d9f7f91cbe2fd196d9cbe8263a0f8f34331fe7b" Oct 30 19:34:48.032791 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.032700 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc4b7dbe508305dfd7caf239d9f7f91cbe2fd196d9cbe8263a0f8f34331fe7b"} err="failed to get container status \"acc4b7dbe508305dfd7caf239d9f7f91cbe2fd196d9cbe8263a0f8f34331fe7b\": rpc error: code = NotFound desc = could not find container \"acc4b7dbe508305dfd7caf239d9f7f91cbe2fd196d9cbe8263a0f8f34331fe7b\": container with ID starting with acc4b7dbe508305dfd7caf239d9f7f91cbe2fd196d9cbe8263a0f8f34331fe7b not found: ID does not exist" Oct 30 19:34:48.032791 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.032720 5047 scope.go:122] "RemoveContainer" containerID="7e8adb1e3556cce603e59051efbb04508134441b93977ceaf6cb443730cbc7fd" Oct 30 19:34:48.045472 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.045454 5047 scope.go:122] "RemoveContainer" containerID="7e8adb1e3556cce603e59051efbb04508134441b93977ceaf6cb443730cbc7fd" Oct 30 19:34:48.045768 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:34:48.045745 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e8adb1e3556cce603e59051efbb04508134441b93977ceaf6cb443730cbc7fd\": container with ID starting with 7e8adb1e3556cce603e59051efbb04508134441b93977ceaf6cb443730cbc7fd not found: ID does not exist" containerID="7e8adb1e3556cce603e59051efbb04508134441b93977ceaf6cb443730cbc7fd" Oct 30 19:34:48.045812 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.045769 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e8adb1e3556cce603e59051efbb04508134441b93977ceaf6cb443730cbc7fd"} err="failed to get container status \"7e8adb1e3556cce603e59051efbb04508134441b93977ceaf6cb443730cbc7fd\": rpc error: code = NotFound desc = could not find container \"7e8adb1e3556cce603e59051efbb04508134441b93977ceaf6cb443730cbc7fd\": container with ID starting with 7e8adb1e3556cce603e59051efbb04508134441b93977ceaf6cb443730cbc7fd not found: ID does not exist" Oct 30 19:34:48.056642 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.056607 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq"] Oct 30 19:34:48.065508 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.065478 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq"] Oct 30 19:34:48.081870 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.081836 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-workload-socket\") pod \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " Oct 30 19:34:48.082160 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.082133 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-workload-socket" pod "10f19076-8c2f-4026-8c4b-ef3062bbe51f" (UID: "10f19076-8c2f-4026-8c4b-ef3062bbe51f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:48.082354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.082324 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-envoy\") pod \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " Oct 30 19:34:48.082814 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.082792 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-token\" (UniqueName: \"kubernetes.io/projected/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-token\") pod \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " Oct 30 19:34:48.083426 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.083400 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istiod-ca-cert" pod "10f19076-8c2f-4026-8c4b-ef3062bbe51f" (UID: "10f19076-8c2f-4026-8c4b-ef3062bbe51f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:48.083555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.083438 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istiod-ca-cert\") pod \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " Oct 30 19:34:48.083555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.083547 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f19076-8c2f-4026-8c4b-ef3062bbe51f-kube-api-access-gn7v8\" (UniqueName: \"kubernetes.io/projected/10f19076-8c2f-4026-8c4b-ef3062bbe51f-kube-api-access-gn7v8\") pod \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " Oct 30 19:34:48.083669 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.083579 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-workload-certs\") pod \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " Oct 30 19:34:48.083669 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.083609 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-podinfo\") pod \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " Oct 30 19:34:48.083776 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.083675 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-data\") pod \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " Oct 30 19:34:48.083776 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.083768 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-credential-socket\") pod \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\" (UID: \"10f19076-8c2f-4026-8c4b-ef3062bbe51f\") " Oct 30 19:34:48.084681 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.084637 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-data" pod "10f19076-8c2f-4026-8c4b-ef3062bbe51f" (UID: "10f19076-8c2f-4026-8c4b-ef3062bbe51f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:48.084770 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.084717 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:48.084770 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.084738 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:48.085166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.085139 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-envoy" pod "10f19076-8c2f-4026-8c4b-ef3062bbe51f" (UID: "10f19076-8c2f-4026-8c4b-ef3062bbe51f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:48.085659 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.085382 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-workload-certs" pod "10f19076-8c2f-4026-8c4b-ef3062bbe51f" (UID: "10f19076-8c2f-4026-8c4b-ef3062bbe51f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:48.085926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.085902 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-credential-socket" pod "10f19076-8c2f-4026-8c4b-ef3062bbe51f" (UID: "10f19076-8c2f-4026-8c4b-ef3062bbe51f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:48.086926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.086897 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h"] Oct 30 19:34:48.087256 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.087231 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f19076-8c2f-4026-8c4b-ef3062bbe51f-kube-api-access-gn7v8" pod "10f19076-8c2f-4026-8c4b-ef3062bbe51f" (UID: "10f19076-8c2f-4026-8c4b-ef3062bbe51f"). InnerVolumeSpecName "kube-api-access-gn7v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:48.087586 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.087563 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-podinfo" pod "10f19076-8c2f-4026-8c4b-ef3062bbe51f" (UID: "10f19076-8c2f-4026-8c4b-ef3062bbe51f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:34:48.088339 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.088313 5047 cpu_manager.go:414] "containerMap: removing container" podUID="0d95b5e1-e759-430f-b0fb-461451ee6e32" containerName="istio-proxy" Oct 30 19:34:48.088416 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.088345 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d95b5e1-e759-430f-b0fb-461451ee6e32" containerName="istio-proxy" Oct 30 19:34:48.088416 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.088395 5047 cpu_manager.go:414] "containerMap: removing container" podUID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" containerName="istio-proxy" Oct 30 19:34:48.088416 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.088404 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" containerName="istio-proxy" Oct 30 19:34:48.088523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.088416 5047 cpu_manager.go:414] "containerMap: removing container" podUID="5c7b1347-f416-42a3-a055-0b866e894c64" containerName="istio-proxy" Oct 30 19:34:48.088523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.088424 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c7b1347-f416-42a3-a055-0b866e894c64" containerName="istio-proxy" Oct 30 19:34:48.088523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.088502 5047 cpu_manager.go:414] "containerMap: removing container" podUID="04defd95-0e95-4ee6-9d47-d6b2488a2f62" containerName="istio-proxy" Oct 30 19:34:48.088523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.088512 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04defd95-0e95-4ee6-9d47-d6b2488a2f62" containerName="istio-proxy" Oct 30 19:34:48.088748 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.088732 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" containerName="istio-proxy" Oct 30 19:34:48.088792 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.088752 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c7b1347-f416-42a3-a055-0b866e894c64" containerName="istio-proxy" Oct 30 19:34:48.088792 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.088777 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" containerName="istio-proxy" Oct 30 19:34:48.088866 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.088793 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="04defd95-0e95-4ee6-9d47-d6b2488a2f62" containerName="istio-proxy" Oct 30 19:34:48.088866 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.088810 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d95b5e1-e759-430f-b0fb-461451ee6e32" containerName="istio-proxy" Oct 30 19:34:48.089488 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.089464 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-token" pod "10f19076-8c2f-4026-8c4b-ef3062bbe51f" (UID: "10f19076-8c2f-4026-8c4b-ef3062bbe51f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:48.093070 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.093048 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.113856 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.109610 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9c8a2a-openshift-default-dockercfg-qsxpt\"" Oct 30 19:34:48.129441 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.129410 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h"] Oct 30 19:34:48.154299 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.153775 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w"] Oct 30 19:34:48.158812 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.158787 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-db77w"] Oct 30 19:34:48.185912 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.185880 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-token\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.186099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.185956 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-podinfo\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.186099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.186059 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-workload-certs\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.186099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.186087 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-credential-socket\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.186289 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.186269 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-data\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.186329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.186319 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ab4a50-e3a3-4312-8de3-4c9656b724de-istiod-ca-cert\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.186377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.186341 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-workload-socket\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.186377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.186363 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l8kv\" (UniqueName: \"kubernetes.io/projected/22ab4a50-e3a3-4312-8de3-4c9656b724de-kube-api-access-4l8kv\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.186542 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.186518 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-envoy\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.186753 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.186738 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:48.186831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.186759 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn7v8\" (UniqueName: \"kubernetes.io/projected/10f19076-8c2f-4026-8c4b-ef3062bbe51f-kube-api-access-gn7v8\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:48.186831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.186773 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:48.186831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.186787 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:48.186831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.186800 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:48.186831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.186816 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:48.186993 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.186832 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f19076-8c2f-4026-8c4b-ef3062bbe51f-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:34:48.195224 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.195192 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv"] Oct 30 19:34:48.199222 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.199188 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-t9rvv"] Oct 30 19:34:48.291044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.290080 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-token\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.291044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.290207 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-podinfo\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.291044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.290251 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-workload-certs\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.291044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.290285 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-credential-socket\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.291044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.290499 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-data\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.291044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.290573 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ab4a50-e3a3-4312-8de3-4c9656b724de-istiod-ca-cert\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.291044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.290614 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-workload-socket\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.291044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.290648 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l8kv\" (UniqueName: \"kubernetes.io/projected/22ab4a50-e3a3-4312-8de3-4c9656b724de-kube-api-access-4l8kv\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.291044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.290703 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-envoy\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.303994 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.296718 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-envoy\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.307625 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.307582 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-data\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.308721 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.307871 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-workload-certs\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.308721 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.308154 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-credential-socket\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.308721 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.308376 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-podinfo\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.308721 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.308427 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-workload-socket\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.310396 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.309093 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ab4a50-e3a3-4312-8de3-4c9656b724de-istiod-ca-cert\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.359159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.359098 5047 patch_prober.go:29] interesting pod/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:34:48.359508 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.359424 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-zbncq" podUID="5c7b1347-f416-42a3-a055-0b866e894c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:34:48.360290 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.359777 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l8kv\" (UniqueName: \"kubernetes.io/projected/22ab4a50-e3a3-4312-8de3-4c9656b724de-kube-api-access-4l8kv\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.373796 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.373727 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-token\") pod \"e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.449250 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.449214 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:48.757380 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.757348 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:48.765633 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:48.765602 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:49.005072 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:34:49.005041 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ab4a50_e3a3_4312_8de3_4c9656b724de.slice/crio-1e225a12c9ecde6d81de9aff57019c518c43d49d7def7715beb78a2cb443a337 WatchSource:0}: Error finding container 1e225a12c9ecde6d81de9aff57019c518c43d49d7def7715beb78a2cb443a337: Status 404 returned error can't find the container with id 1e225a12c9ecde6d81de9aff57019c518c43d49d7def7715beb78a2cb443a337 Oct 30 19:34:49.007308 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.007285 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h"] Oct 30 19:34:49.008499 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.008454 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:49.008565 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.008548 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:49.008608 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.008594 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:49.010981 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.010958 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" Oct 30 19:34:49.010981 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.010964 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d" event={"ID":"10f19076-8c2f-4026-8c4b-ef3062bbe51f","Type":"ContainerDied","Data":"4131ab6faf6b159c39a768891370e829587da882d8a2a11559c72ef36945c72e"} Oct 30 19:34:49.011145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.011037 5047 scope.go:122] "RemoveContainer" containerID="30226ee388745ce9ca41c8ccde0c1d49eaf56bc89387a7df0abbf1aa3a6ce660" Oct 30 19:34:49.016154 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.016129 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:49.017189 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.017168 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:34:49.050942 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.050914 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04defd95-0e95-4ee6-9d47-d6b2488a2f62" path="/var/lib/kubelet/pods/04defd95-0e95-4ee6-9d47-d6b2488a2f62/volumes" Oct 30 19:34:49.051581 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.051559 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d95b5e1-e759-430f-b0fb-461451ee6e32" path="/var/lib/kubelet/pods/0d95b5e1-e759-430f-b0fb-461451ee6e32/volumes" Oct 30 19:34:49.052129 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.052110 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c7b1347-f416-42a3-a055-0b866e894c64" path="/var/lib/kubelet/pods/5c7b1347-f416-42a3-a055-0b866e894c64/volumes" Oct 30 19:34:49.197411 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.197374 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d"] Oct 30 19:34:49.209674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.209644 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c5449c-openshift-default-6847484469-w5w2d"] Oct 30 19:34:49.626077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.626041 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd"] Oct 30 19:34:49.626913 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.626897 5047 cpu_manager.go:414] "containerMap: removing container" podUID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" containerName="istio-proxy" Oct 30 19:34:49.626963 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.626915 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" containerName="istio-proxy" Oct 30 19:34:49.629804 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.629787 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:34:49.661869 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.661834 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-7d9e82-serving-cert\"" Oct 30 19:34:49.671773 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.671732 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd"] Oct 30 19:34:49.809730 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.809690 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edca9e01-ae45-430c-8478-07d1667a7c07-maas-api-tls\") pod \"maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd\" (UID: \"edca9e01-ae45-430c-8478-07d1667a7c07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:34:49.809919 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.809827 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2cqv\" (UniqueName: \"kubernetes.io/projected/edca9e01-ae45-430c-8478-07d1667a7c07-kube-api-access-l2cqv\") pod \"maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd\" (UID: \"edca9e01-ae45-430c-8478-07d1667a7c07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:34:49.809979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.809963 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edca9e01-ae45-430c-8478-07d1667a7c07-metrics-certs\") pod \"maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd\" (UID: \"edca9e01-ae45-430c-8478-07d1667a7c07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:34:49.810077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.810060 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edca9e01-ae45-430c-8478-07d1667a7c07-openshift-service-ca\") pod \"maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd\" (UID: \"edca9e01-ae45-430c-8478-07d1667a7c07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:34:49.999572 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.991335 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edca9e01-ae45-430c-8478-07d1667a7c07-maas-api-tls\") pod \"maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd\" (UID: \"edca9e01-ae45-430c-8478-07d1667a7c07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:34:49.999572 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.991471 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2cqv\" (UniqueName: \"kubernetes.io/projected/edca9e01-ae45-430c-8478-07d1667a7c07-kube-api-access-l2cqv\") pod \"maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd\" (UID: \"edca9e01-ae45-430c-8478-07d1667a7c07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:34:49.999572 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.991618 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edca9e01-ae45-430c-8478-07d1667a7c07-metrics-certs\") pod \"maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd\" (UID: \"edca9e01-ae45-430c-8478-07d1667a7c07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:34:49.999572 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.991712 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edca9e01-ae45-430c-8478-07d1667a7c07-openshift-service-ca\") pod \"maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd\" (UID: \"edca9e01-ae45-430c-8478-07d1667a7c07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:34:49.999572 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:49.992694 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edca9e01-ae45-430c-8478-07d1667a7c07-openshift-service-ca\") pod \"maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd\" (UID: \"edca9e01-ae45-430c-8478-07d1667a7c07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:34:50.040310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:50.040258 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edca9e01-ae45-430c-8478-07d1667a7c07-metrics-certs\") pod \"maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd\" (UID: \"edca9e01-ae45-430c-8478-07d1667a7c07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:34:50.044418 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:50.044097 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edca9e01-ae45-430c-8478-07d1667a7c07-maas-api-tls\") pod \"maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd\" (UID: \"edca9e01-ae45-430c-8478-07d1667a7c07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:34:50.055275 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:50.055130 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2cqv\" (UniqueName: \"kubernetes.io/projected/edca9e01-ae45-430c-8478-07d1667a7c07-kube-api-access-l2cqv\") pod \"maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd\" (UID: \"edca9e01-ae45-430c-8478-07d1667a7c07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:34:50.071166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:50.071125 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" event={"ID":"22ab4a50-e3a3-4312-8de3-4c9656b724de","Type":"ContainerStarted","Data":"d3a04500d82cb2b769d0cfe5acc230928b4320dca934df79bf853ee2b17aaaae"} Oct 30 19:34:50.071411 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:50.071390 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" event={"ID":"22ab4a50-e3a3-4312-8de3-4c9656b724de","Type":"ContainerStarted","Data":"1e225a12c9ecde6d81de9aff57019c518c43d49d7def7715beb78a2cb443a337"} Oct 30 19:34:50.117423 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:50.117370 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" podStartSLOduration=2.117350758 podStartE2EDuration="2.117350758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:50.113453654 +0000 UTC m=+1843.187909014" watchObservedRunningTime="2026-10-30 19:34:50.117350758 +0000 UTC m=+1843.191806117" Oct 30 19:34:50.258839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:50.257844 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:34:50.450813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:50.450058 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:50.457075 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:50.457046 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:50.663275 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:34:50.663242 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedca9e01_ae45_430c_8478_07d1667a7c07.slice/crio-d227716ad5aaeca4e850ee4d974606ad864e647ca48e5a9881d8909d03c3348e WatchSource:0}: Error finding container d227716ad5aaeca4e850ee4d974606ad864e647ca48e5a9881d8909d03c3348e: Status 404 returned error can't find the container with id d227716ad5aaeca4e850ee4d974606ad864e647ca48e5a9881d8909d03c3348e Oct 30 19:34:50.666234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:50.666208 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd"] Oct 30 19:34:51.179383 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:51.175285 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f19076-8c2f-4026-8c4b-ef3062bbe51f" path="/var/lib/kubelet/pods/10f19076-8c2f-4026-8c4b-ef3062bbe51f/volumes" Oct 30 19:34:51.179383 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:51.175842 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" event={"ID":"edca9e01-ae45-430c-8478-07d1667a7c07","Type":"ContainerStarted","Data":"d227716ad5aaeca4e850ee4d974606ad864e647ca48e5a9881d8909d03c3348e"} Oct 30 19:34:51.179383 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:51.175903 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:51.179383 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:51.175974 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:34:52.128711 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:52.127981 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" event={"ID":"edca9e01-ae45-430c-8478-07d1667a7c07","Type":"ContainerStarted","Data":"5bbcb82b1db33924aca6539b364b150383b695559abf7eacb616bafd00a3a674"} Oct 30 19:34:52.128711 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:52.128460 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:34:52.204055 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:52.192827 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" podStartSLOduration=2.721839762 podStartE2EDuration="3.192805596s" totalImagesPullingTime="470.965834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:52.191503333 +0000 UTC m=+1845.265958693" watchObservedRunningTime="2026-10-30 19:34:52.192805596 +0000 UTC m=+1845.267260959" Oct 30 19:34:55.416602 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:55.416560 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk"] Oct 30 19:34:55.421404 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:55.421380 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" Oct 30 19:34:55.439658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:55.439623 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk"] Oct 30 19:34:55.444869 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:55.444838 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9c8a2a\"" Oct 30 19:34:55.444869 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:55.444838 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9c8a2a-dockercfg-zm7tt\"" Oct 30 19:34:55.522034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:55.519350 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd8e9c4-fdaa-45ff-a214-cac92b05c139-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk\" (UID: \"dbd8e9c4-fdaa-45ff-a214-cac92b05c139\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" Oct 30 19:34:55.522034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:55.519669 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x8q5\" (UniqueName: \"kubernetes.io/projected/dbd8e9c4-fdaa-45ff-a214-cac92b05c139-kube-api-access-5x8q5\") pod \"payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk\" (UID: \"dbd8e9c4-fdaa-45ff-a214-cac92b05c139\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" Oct 30 19:34:55.634712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:55.634644 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x8q5\" (UniqueName: \"kubernetes.io/projected/dbd8e9c4-fdaa-45ff-a214-cac92b05c139-kube-api-access-5x8q5\") pod \"payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk\" (UID: \"dbd8e9c4-fdaa-45ff-a214-cac92b05c139\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" Oct 30 19:34:55.635962 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:55.635929 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd8e9c4-fdaa-45ff-a214-cac92b05c139-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk\" (UID: \"dbd8e9c4-fdaa-45ff-a214-cac92b05c139\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" Oct 30 19:34:55.638745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:55.637981 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd8e9c4-fdaa-45ff-a214-cac92b05c139-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk\" (UID: \"dbd8e9c4-fdaa-45ff-a214-cac92b05c139\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" Oct 30 19:34:55.715413 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:55.715322 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x8q5\" (UniqueName: \"kubernetes.io/projected/dbd8e9c4-fdaa-45ff-a214-cac92b05c139-kube-api-access-5x8q5\") pod \"payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk\" (UID: \"dbd8e9c4-fdaa-45ff-a214-cac92b05c139\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" Oct 30 19:34:55.751134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:55.743367 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" Oct 30 19:34:56.192826 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:34:56.192794 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd8e9c4_fdaa_45ff_a214_cac92b05c139.slice/crio-e2560a2920f2b79fb10c2efe35969a164be7aeacb3dbeccffda84ed3b96e3878 WatchSource:0}: Error finding container e2560a2920f2b79fb10c2efe35969a164be7aeacb3dbeccffda84ed3b96e3878: Status 404 returned error can't find the container with id e2560a2920f2b79fb10c2efe35969a164be7aeacb3dbeccffda84ed3b96e3878 Oct 30 19:34:56.194170 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:56.194143 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk"] Oct 30 19:34:56.214371 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:56.214336 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" event={"ID":"dbd8e9c4-fdaa-45ff-a214-cac92b05c139","Type":"ContainerStarted","Data":"e2560a2920f2b79fb10c2efe35969a164be7aeacb3dbeccffda84ed3b96e3878"} Oct 30 19:34:57.224461 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:57.224426 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" event={"ID":"dbd8e9c4-fdaa-45ff-a214-cac92b05c139","Type":"ContainerStarted","Data":"5f8c95daa048950bd30fd9344865277a92405977e82aec179b0ddea0259fa0ce"} Oct 30 19:34:57.224873 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:57.224573 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" Oct 30 19:34:57.263650 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:57.263581 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" podStartSLOduration=2.263559839 podStartE2EDuration="2.263559839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:57.261600715 +0000 UTC m=+1850.336056076" watchObservedRunningTime="2026-10-30 19:34:57.263559839 +0000 UTC m=+1850.338015199" Oct 30 19:34:58.147730 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:34:58.147695 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:35:00.166281 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.166237 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf"] Oct 30 19:35:00.171617 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.171591 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 19:35:00.189811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.189774 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf"] Oct 30 19:35:00.196350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.196313 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-287ea2-serving-cert\"" Oct 30 19:35:00.273460 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.273391 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b442c9-df9e-42fb-a944-5b1ac933efa4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf\" (UID: \"89b442c9-df9e-42fb-a944-5b1ac933efa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 19:35:00.273679 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.273465 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plqpw\" (UniqueName: \"kubernetes.io/projected/89b442c9-df9e-42fb-a944-5b1ac933efa4-kube-api-access-plqpw\") pod \"maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf\" (UID: \"89b442c9-df9e-42fb-a944-5b1ac933efa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 19:35:00.273769 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.273734 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b442c9-df9e-42fb-a944-5b1ac933efa4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf\" (UID: \"89b442c9-df9e-42fb-a944-5b1ac933efa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 19:35:00.273910 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.273824 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b442c9-df9e-42fb-a944-5b1ac933efa4-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf\" (UID: \"89b442c9-df9e-42fb-a944-5b1ac933efa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 19:35:00.375108 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.375065 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b442c9-df9e-42fb-a944-5b1ac933efa4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf\" (UID: \"89b442c9-df9e-42fb-a944-5b1ac933efa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 19:35:00.375274 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.375136 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plqpw\" (UniqueName: \"kubernetes.io/projected/89b442c9-df9e-42fb-a944-5b1ac933efa4-kube-api-access-plqpw\") pod \"maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf\" (UID: \"89b442c9-df9e-42fb-a944-5b1ac933efa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 19:35:00.375390 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.375282 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b442c9-df9e-42fb-a944-5b1ac933efa4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf\" (UID: \"89b442c9-df9e-42fb-a944-5b1ac933efa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 19:35:00.375390 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.375360 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b442c9-df9e-42fb-a944-5b1ac933efa4-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf\" (UID: \"89b442c9-df9e-42fb-a944-5b1ac933efa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 19:35:00.376145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.376119 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b442c9-df9e-42fb-a944-5b1ac933efa4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf\" (UID: \"89b442c9-df9e-42fb-a944-5b1ac933efa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 19:35:00.377959 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.377923 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b442c9-df9e-42fb-a944-5b1ac933efa4-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf\" (UID: \"89b442c9-df9e-42fb-a944-5b1ac933efa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 19:35:00.378210 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.378188 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b442c9-df9e-42fb-a944-5b1ac933efa4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf\" (UID: \"89b442c9-df9e-42fb-a944-5b1ac933efa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 19:35:00.420095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.417204 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plqpw\" (UniqueName: \"kubernetes.io/projected/89b442c9-df9e-42fb-a944-5b1ac933efa4-kube-api-access-plqpw\") pod \"maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf\" (UID: \"89b442c9-df9e-42fb-a944-5b1ac933efa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 19:35:00.492498 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.492448 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 19:35:00.838601 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.838561 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww"] Oct 30 19:35:00.883413 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:35:00.883379 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b442c9_df9e_42fb_a944_5b1ac933efa4.slice/crio-d6c5ef521bb1ecef7c7b6b518db359fd715d57ed72d74f45e16710263cce42d3 WatchSource:0}: Error finding container d6c5ef521bb1ecef7c7b6b518db359fd715d57ed72d74f45e16710263cce42d3: Status 404 returned error can't find the container with id d6c5ef521bb1ecef7c7b6b518db359fd715d57ed72d74f45e16710263cce42d3 Oct 30 19:35:00.885589 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:00.885556 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf"] Oct 30 19:35:01.312575 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:01.309293 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" event={"ID":"89b442c9-df9e-42fb-a944-5b1ac933efa4","Type":"ContainerStarted","Data":"d6c5ef521bb1ecef7c7b6b518db359fd715d57ed72d74f45e16710263cce42d3"} Oct 30 19:35:02.323829 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:02.323783 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" event={"ID":"89b442c9-df9e-42fb-a944-5b1ac933efa4","Type":"ContainerStarted","Data":"954651eb219e4a8260454c4cf9275a13a926b58e42b3f0d6bee467ada055b146"} Oct 30 19:35:02.324327 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:02.323887 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 19:35:02.382397 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:02.382338 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" podStartSLOduration=1.9664570449999998 podStartE2EDuration="2.382321566s" totalImagesPullingTime="415.864521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:02.36626358 +0000 UTC m=+1855.440718939" watchObservedRunningTime="2026-10-30 19:35:02.382321566 +0000 UTC m=+1855.456776921" Oct 30 19:35:02.387515 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:02.387479 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg"] Oct 30 19:35:02.387736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:02.387711 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" containerName="istio-proxy" containerID="cri-o://42d44f42b473a9756449131fd12d14aafddc51b3e4764864c0084e4c208d1229" gracePeriod=30 Oct 30 19:35:05.839399 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:05.839346 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" podUID="820cab28-9ecc-46d5-945a-df0b07cf3c4f" containerName="payload-pre-processing" containerID="cri-o://d6477213fc05d20752053e97a964079703e71469b543977a830a37e10d04808d" gracePeriod=25 Oct 30 19:35:07.707199 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.707169 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:35:07.806198 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.806160 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-token\" (UniqueName: \"kubernetes.io/projected/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-token\") pod \"c080477b-5afb-4972-8feb-1bd6ec66092f\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " Oct 30 19:35:07.806427 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.806395 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-data\") pod \"c080477b-5afb-4972-8feb-1bd6ec66092f\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " Oct 30 19:35:07.806502 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.806428 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-podinfo\") pod \"c080477b-5afb-4972-8feb-1bd6ec66092f\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " Oct 30 19:35:07.806502 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.806476 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-credential-socket\") pod \"c080477b-5afb-4972-8feb-1bd6ec66092f\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " Oct 30 19:35:07.806616 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.806510 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c080477b-5afb-4972-8feb-1bd6ec66092f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c080477b-5afb-4972-8feb-1bd6ec66092f-istiod-ca-cert\") pod \"c080477b-5afb-4972-8feb-1bd6ec66092f\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " Oct 30 19:35:07.806616 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.806585 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c080477b-5afb-4972-8feb-1bd6ec66092f-kube-api-access-ppz85\" (UniqueName: \"kubernetes.io/projected/c080477b-5afb-4972-8feb-1bd6ec66092f-kube-api-access-ppz85\") pod \"c080477b-5afb-4972-8feb-1bd6ec66092f\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " Oct 30 19:35:07.806718 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.806696 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-workload-socket\") pod \"c080477b-5afb-4972-8feb-1bd6ec66092f\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " Oct 30 19:35:07.806780 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.806759 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-workload-certs\") pod \"c080477b-5afb-4972-8feb-1bd6ec66092f\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " Oct 30 19:35:07.806835 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.806792 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-data" pod "c080477b-5afb-4972-8feb-1bd6ec66092f" (UID: "c080477b-5afb-4972-8feb-1bd6ec66092f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:07.806964 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.806899 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-credential-socket" pod "c080477b-5afb-4972-8feb-1bd6ec66092f" (UID: "c080477b-5afb-4972-8feb-1bd6ec66092f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:07.807186 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.807000 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-workload-certs" pod "c080477b-5afb-4972-8feb-1bd6ec66092f" (UID: "c080477b-5afb-4972-8feb-1bd6ec66092f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:07.807321 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.807062 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-envoy\") pod \"c080477b-5afb-4972-8feb-1bd6ec66092f\" (UID: \"c080477b-5afb-4972-8feb-1bd6ec66092f\") " Oct 30 19:35:07.807416 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.807242 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-workload-socket" pod "c080477b-5afb-4972-8feb-1bd6ec66092f" (UID: "c080477b-5afb-4972-8feb-1bd6ec66092f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:07.807785 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.807735 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c080477b-5afb-4972-8feb-1bd6ec66092f-istiod-ca-cert" pod "c080477b-5afb-4972-8feb-1bd6ec66092f" (UID: "c080477b-5afb-4972-8feb-1bd6ec66092f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:07.808385 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.808357 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:07.808483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.808390 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:07.808483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.808405 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c080477b-5afb-4972-8feb-1bd6ec66092f-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:07.808483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.808417 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:07.808483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.808429 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:07.809310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.809285 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-podinfo" pod "c080477b-5afb-4972-8feb-1bd6ec66092f" (UID: "c080477b-5afb-4972-8feb-1bd6ec66092f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:35:07.810111 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.810090 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-envoy" pod "c080477b-5afb-4972-8feb-1bd6ec66092f" (UID: "c080477b-5afb-4972-8feb-1bd6ec66092f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:07.810452 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.810435 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-token" pod "c080477b-5afb-4972-8feb-1bd6ec66092f" (UID: "c080477b-5afb-4972-8feb-1bd6ec66092f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:07.810606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.810590 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c080477b-5afb-4972-8feb-1bd6ec66092f-kube-api-access-ppz85" pod "c080477b-5afb-4972-8feb-1bd6ec66092f" (UID: "c080477b-5afb-4972-8feb-1bd6ec66092f"). InnerVolumeSpecName "kube-api-access-ppz85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:07.910087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.910036 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:07.910087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.910072 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:07.910087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.910081 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c080477b-5afb-4972-8feb-1bd6ec66092f-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:07.910087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:07.910091 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppz85\" (UniqueName: \"kubernetes.io/projected/c080477b-5afb-4972-8feb-1bd6ec66092f-kube-api-access-ppz85\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:08.178357 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:08.172250 5047 scope.go:122] "RemoveContainer" containerID="42d44f42b473a9756449131fd12d14aafddc51b3e4764864c0084e4c208d1229" Oct 30 19:35:08.302557 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:08.302434 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" Oct 30 19:35:08.345034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:08.344313 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 19:35:08.937029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:08.930650 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" event={"ID":"c080477b-5afb-4972-8feb-1bd6ec66092f","Type":"ContainerDied","Data":"42d44f42b473a9756449131fd12d14aafddc51b3e4764864c0084e4c208d1229"} Oct 30 19:35:08.937029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:08.930732 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" event={"ID":"c080477b-5afb-4972-8feb-1bd6ec66092f","Type":"ContainerDied","Data":"0ce41d883babf8f481e8f2837f685e11cc5709b93fdf75be1384130c6bf3b4ba"} Oct 30 19:35:08.937029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:08.930810 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg" Oct 30 19:35:09.614334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:09.614298 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg"] Oct 30 19:35:09.873089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:09.872483 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-qb8qg"] Oct 30 19:35:10.045061 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:35:10.033176 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:35:10.455033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:10.442654 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" podUID="c935589c-ed56-4012-85fd-b628a0e50980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:10.476038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:10.475544 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:11.040438 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:10.824730 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 19:35:11.040438 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:35:10.835982 5047 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc080477b_5afb_4972_8feb_1bd6ec66092f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc080477b_5afb_4972_8feb_1bd6ec66092f.slice/crio-0ce41d883babf8f481e8f2837f685e11cc5709b93fdf75be1384130c6bf3b4ba\": RecentStats: unable to find data in memory cache]" Oct 30 19:35:11.091471 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:11.091419 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 30 19:35:11.176034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:11.127562 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Oct 30 19:35:11.176034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:11.164537 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:35:11.460031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:11.447157 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" podUID="c935589c-ed56-4012-85fd-b628a0e50980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:11.477038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:11.476641 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:11.545035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:11.525923 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" path="/var/lib/kubelet/pods/c080477b-5afb-4972-8feb-1bd6ec66092f/volumes" Oct 30 19:35:11.545035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:11.526537 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:35:11.545035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:11.526595 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:35:11.851031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:11.830100 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 19:35:11.851031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:11.831223 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:35:12.101143 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:12.093269 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:12.101143 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:12.094064 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:35:12.153085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:12.136099 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:12.200031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:12.192281 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:35:12.387491 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:12.387403 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:35:12.387742 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:12.387723 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:35:12.499031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:12.476440 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" podUID="c935589c-ed56-4012-85fd-b628a0e50980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded" Oct 30 19:35:12.499031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:12.476837 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:12.675085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:12.671560 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:12.783028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:12.764602 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:35:12.806031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:12.804840 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:35:12.816447 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:12.816257 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:35:15.807163 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:15.807120 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" podUID="820cab28-9ecc-46d5-945a-df0b07cf3c4f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:35:17.430459 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:17.430422 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j"] Oct 30 19:35:17.431065 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:17.430621 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" podUID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" containerName="istio-proxy" containerID="cri-o://343cbb955797d7758c8589372d766e6cdc279a36fb8dee3c78c5b9efb04b8c66" gracePeriod=30 Oct 30 19:35:20.481237 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:20.481186 5047 patch_prober.go:29] interesting pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:35:20.481727 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:20.481268 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" podUID="1713eb56-d041-40ae-824a-3bdc1ca843f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:21.737473 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:21.737435 5047 generic.go:356] "Generic (PLEG): container finished" podID="1713eb56-d041-40ae-824a-3bdc1ca843f9" containerID="501f90f756a7578030e096d0b1936d121121055b50a97aff3e43ae496d9a223f" exitCode=0 Oct 30 19:35:21.739931 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:21.737510 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" event={"ID":"1713eb56-d041-40ae-824a-3bdc1ca843f9","Type":"ContainerDied","Data":"501f90f756a7578030e096d0b1936d121121055b50a97aff3e43ae496d9a223f"} Oct 30 19:35:21.739931 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:21.738042 5047 scope.go:122] "RemoveContainer" containerID="501f90f756a7578030e096d0b1936d121121055b50a97aff3e43ae496d9a223f" Oct 30 19:35:21.740516 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:21.739966 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:21.740516 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:21.740053 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:21.740516 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:21.740088 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:22.766901 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:22.766813 5047 generic.go:356] "Generic (PLEG): container finished" podID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" containerID="343cbb955797d7758c8589372d766e6cdc279a36fb8dee3c78c5b9efb04b8c66" exitCode=0 Oct 30 19:35:22.766901 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:22.766886 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" event={"ID":"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75","Type":"ContainerDied","Data":"343cbb955797d7758c8589372d766e6cdc279a36fb8dee3c78c5b9efb04b8c66"} Oct 30 19:35:22.769137 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:22.769112 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" event={"ID":"1713eb56-d041-40ae-824a-3bdc1ca843f9","Type":"ContainerStarted","Data":"e631a021dd9bb5db2053ea0f3b362a219e96ea09c0532318483bbf4dc8be2baa"} Oct 30 19:35:22.770822 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:22.770802 5047 generic.go:356] "Generic (PLEG): container finished" podID="5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" containerID="ee8d8c04f7a511e00231382c4cbba30d2e7ff086f1260edcd3dfe2016dc256db" exitCode=0 Oct 30 19:35:22.770959 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:22.770859 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" event={"ID":"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96","Type":"ContainerDied","Data":"ee8d8c04f7a511e00231382c4cbba30d2e7ff086f1260edcd3dfe2016dc256db"} Oct 30 19:35:22.771360 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:22.771343 5047 scope.go:122] "RemoveContainer" containerID="ee8d8c04f7a511e00231382c4cbba30d2e7ff086f1260edcd3dfe2016dc256db" Oct 30 19:35:22.773390 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:22.773354 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:22.773554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:22.773454 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:22.773646 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:22.773585 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:23.155624 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:23.155374 5047 patch_prober.go:29] interesting pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 19:35:23.162654 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:23.156370 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" podUID="1713eb56-d041-40ae-824a-3bdc1ca843f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 19:35:23.186555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:23.186502 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:35:23.801533 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:23.801087 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" event={"ID":"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96","Type":"ContainerStarted","Data":"9dba553322c262ad1953012029f6be119177a7b2b7482c6aaabe1a1d4c8e8103"} Oct 30 19:35:24.003054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.001745 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:35:24.030034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.025250 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:35:24.133707 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.111746 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-envoy\") pod \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " Oct 30 19:35:24.134101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.134074 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-credential-socket\") pod \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " Oct 30 19:35:24.134639 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.134617 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-workload-certs\") pod \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " Oct 30 19:35:24.135110 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.135090 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-token\" (UniqueName: \"kubernetes.io/projected/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-token\") pod \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " Oct 30 19:35:24.135901 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.135876 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-data\") pod \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " Oct 30 19:35:24.136509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.136462 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-podinfo\") pod \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " Oct 30 19:35:24.137033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.136998 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istiod-ca-cert\") pod \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " Oct 30 19:35:24.137386 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.137309 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-kube-api-access-84qtt\" (UniqueName: \"kubernetes.io/projected/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-kube-api-access-84qtt\") pod \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " Oct 30 19:35:24.149038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.134541 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-credential-socket" pod "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" (UID: "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:24.149038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.134946 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-workload-certs" pod "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" (UID: "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:24.149038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.136338 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-data" pod "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" (UID: "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:24.149038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.139399 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istiod-ca-cert" pod "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" (UID: "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:24.149038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.148214 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-token" pod "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" (UID: "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:24.149578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.149554 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-workload-socket\") pod \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\" (UID: \"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75\") " Oct 30 19:35:24.150668 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.150644 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:24.150796 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.150784 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:24.150877 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.150865 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:24.150977 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.150967 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:24.151088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.151076 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:24.151955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.151901 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-workload-socket" pod "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" (UID: "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:24.182716 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.182471 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-podinfo" pod "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" (UID: "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:35:24.182716 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.182583 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-kube-api-access-84qtt" pod "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" (UID: "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75"). InnerVolumeSpecName "kube-api-access-84qtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:24.182716 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.182666 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-envoy" pod "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" (UID: "76c276d2-2b71-4701-a4f2-bf3dbc9f9a75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:24.182953 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.182757 5047 patch_prober.go:29] interesting pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 19:35:24.182953 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.182804 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" podUID="1713eb56-d041-40ae-824a-3bdc1ca843f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 19:35:24.278031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.272516 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84qtt\" (UniqueName: \"kubernetes.io/projected/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-kube-api-access-84qtt\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:24.278031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.272562 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:24.278031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.272579 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:24.278031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.272593 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:24.788033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.786286 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:35:24.788033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.787137 5047 patch_prober.go:29] interesting pod/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 19:35:24.788033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.787191 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" podUID="5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 19:35:24.812202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.812169 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" Oct 30 19:35:24.812202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.812190 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j" event={"ID":"76c276d2-2b71-4701-a4f2-bf3dbc9f9a75","Type":"ContainerDied","Data":"7b42125b7b65cc213bf77ff15c39de10490e18f6ec608251b2538c04ed03a605"} Oct 30 19:35:24.812806 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.812248 5047 scope.go:122] "RemoveContainer" containerID="343cbb955797d7758c8589372d766e6cdc279a36fb8dee3c78c5b9efb04b8c66" Oct 30 19:35:24.899377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.899335 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j"] Oct 30 19:35:24.903494 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:24.903465 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-gzt5j"] Oct 30 19:35:25.097423 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:25.097387 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" path="/var/lib/kubelet/pods/76c276d2-2b71-4701-a4f2-bf3dbc9f9a75/volumes" Oct 30 19:35:25.171029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:25.163564 5047 patch_prober.go:29] interesting pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 19:35:25.171029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:25.163633 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" podUID="1713eb56-d041-40ae-824a-3bdc1ca843f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 19:35:25.782030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:25.781420 5047 patch_prober.go:29] interesting pod/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 19:35:25.782030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:25.781498 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" podUID="5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 19:35:25.806697 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:25.806658 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" podUID="820cab28-9ecc-46d5-945a-df0b07cf3c4f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:35:26.158536 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.158501 5047 patch_prober.go:29] interesting pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 19:35:26.159148 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.159113 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" podUID="1713eb56-d041-40ae-824a-3bdc1ca843f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 19:35:26.273030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.259228 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" Oct 30 19:35:26.400686 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.400649 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/820cab28-9ecc-46d5-945a-df0b07cf3c4f-kube-api-access-v8xdd\" (UniqueName: \"kubernetes.io/projected/820cab28-9ecc-46d5-945a-df0b07cf3c4f-kube-api-access-v8xdd\") pod \"820cab28-9ecc-46d5-945a-df0b07cf3c4f\" (UID: \"820cab28-9ecc-46d5-945a-df0b07cf3c4f\") " Oct 30 19:35:26.400880 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.400868 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/820cab28-9ecc-46d5-945a-df0b07cf3c4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820cab28-9ecc-46d5-945a-df0b07cf3c4f-plugins-config-volume\") pod \"820cab28-9ecc-46d5-945a-df0b07cf3c4f\" (UID: \"820cab28-9ecc-46d5-945a-df0b07cf3c4f\") " Oct 30 19:35:26.401603 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.401582 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/820cab28-9ecc-46d5-945a-df0b07cf3c4f-plugins-config-volume" pod "820cab28-9ecc-46d5-945a-df0b07cf3c4f" (UID: "820cab28-9ecc-46d5-945a-df0b07cf3c4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:26.403663 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.403631 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820cab28-9ecc-46d5-945a-df0b07cf3c4f-kube-api-access-v8xdd" pod "820cab28-9ecc-46d5-945a-df0b07cf3c4f" (UID: "820cab28-9ecc-46d5-945a-df0b07cf3c4f"). InnerVolumeSpecName "kube-api-access-v8xdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:26.502776 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.502736 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8xdd\" (UniqueName: \"kubernetes.io/projected/820cab28-9ecc-46d5-945a-df0b07cf3c4f-kube-api-access-v8xdd\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:26.502776 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.502776 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820cab28-9ecc-46d5-945a-df0b07cf3c4f-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:26.783090 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.782433 5047 patch_prober.go:29] interesting pod/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 19:35:26.783090 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.782498 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" podUID="5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 19:35:26.851904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.850332 5047 generic.go:356] "Generic (PLEG): container finished" podID="820cab28-9ecc-46d5-945a-df0b07cf3c4f" containerID="d6477213fc05d20752053e97a964079703e71469b543977a830a37e10d04808d" exitCode=0 Oct 30 19:35:26.851904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.850631 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" event={"ID":"820cab28-9ecc-46d5-945a-df0b07cf3c4f","Type":"ContainerDied","Data":"d6477213fc05d20752053e97a964079703e71469b543977a830a37e10d04808d"} Oct 30 19:35:26.851904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.850690 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" event={"ID":"820cab28-9ecc-46d5-945a-df0b07cf3c4f","Type":"ContainerDied","Data":"733e7a12d01adb799e0b3cf8ffa88a482470f9a13a42db692baed5d2085beea0"} Oct 30 19:35:26.851904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.850723 5047 scope.go:122] "RemoveContainer" containerID="d6477213fc05d20752053e97a964079703e71469b543977a830a37e10d04808d" Oct 30 19:35:26.851904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.850889 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww" Oct 30 19:35:26.921184 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.921065 5047 scope.go:122] "RemoveContainer" containerID="d6477213fc05d20752053e97a964079703e71469b543977a830a37e10d04808d" Oct 30 19:35:26.921491 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:35:26.921460 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6477213fc05d20752053e97a964079703e71469b543977a830a37e10d04808d\": container with ID starting with d6477213fc05d20752053e97a964079703e71469b543977a830a37e10d04808d not found: ID does not exist" containerID="d6477213fc05d20752053e97a964079703e71469b543977a830a37e10d04808d" Oct 30 19:35:26.921576 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:26.921506 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6477213fc05d20752053e97a964079703e71469b543977a830a37e10d04808d"} err="failed to get container status \"d6477213fc05d20752053e97a964079703e71469b543977a830a37e10d04808d\": rpc error: code = NotFound desc = could not find container \"d6477213fc05d20752053e97a964079703e71469b543977a830a37e10d04808d\": container with ID starting with d6477213fc05d20752053e97a964079703e71469b543977a830a37e10d04808d not found: ID does not exist" Oct 30 19:35:27.001845 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:27.001749 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww"] Oct 30 19:35:27.212039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:27.207318 5047 patch_prober.go:29] interesting pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 19:35:27.212039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:27.207455 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" podUID="1713eb56-d041-40ae-824a-3bdc1ca843f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 19:35:27.223379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:27.222416 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-skrww"] Oct 30 19:35:27.785033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:27.781409 5047 patch_prober.go:29] interesting pod/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 19:35:27.785033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:27.781502 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" podUID="5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 19:35:28.164031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:28.160281 5047 patch_prober.go:29] interesting pod/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 19:35:28.164031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:28.160344 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" podUID="1713eb56-d041-40ae-824a-3bdc1ca843f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 19:35:28.795062 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:28.783233 5047 patch_prober.go:29] interesting pod/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 19:35:28.795062 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:28.783315 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" podUID="5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 19:35:29.076772 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:29.073279 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="820cab28-9ecc-46d5-945a-df0b07cf3c4f" path="/var/lib/kubelet/pods/820cab28-9ecc-46d5-945a-df0b07cf3c4f/volumes" Oct 30 19:35:29.164696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:29.164641 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:35:29.165384 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:29.165360 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:35:29.167300 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:29.167279 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:35:29.781435 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:29.781393 5047 patch_prober.go:29] interesting pod/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 19:35:29.781624 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:29.781477 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" podUID="5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 19:35:30.786223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:30.786191 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:35:30.790086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:30.790055 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:35:32.294708 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:32.294667 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt"] Oct 30 19:35:32.295159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:32.294925 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" podUID="407464c4-7b3c-4c02-9bde-db3e9a0e97a7" containerName="istio-proxy" containerID="cri-o://339aaa9d151381de549fbc0d1204d9c8fab4c0de6a8ce7c8692c5223eb0db804" gracePeriod=30 Oct 30 19:35:32.641744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:32.641703 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c"] Oct 30 19:35:32.641977 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:32.641954 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" podUID="d0e81adf-2dbf-41ff-a413-13ddc4310c6c" containerName="istio-proxy" containerID="cri-o://66548573291e74450d87f4fded492f635601cf85b59058f1672766cc7948a7fc" gracePeriod=30 Oct 30 19:35:32.646398 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:32.646365 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj"] Oct 30 19:35:32.646590 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:32.646570 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" podUID="9399ee4d-b327-4739-85b1-d18eecea8116" containerName="istio-proxy" containerID="cri-o://5437db989a834dc6349d6ce5a9c5cf7ba10752132fa498067855c29f51b0380b" gracePeriod=30 Oct 30 19:35:37.627984 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.627950 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:35:37.674816 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.674784 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-credential-socket\") pod \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " Oct 30 19:35:37.674948 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.674852 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-workload-socket\") pod \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " Oct 30 19:35:37.674948 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.674942 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-envoy\") pod \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " Oct 30 19:35:37.675077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.674982 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-data\") pod \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " Oct 30 19:35:37.675077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.675053 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istiod-ca-cert\") pod \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " Oct 30 19:35:37.675159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.675067 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-credential-socket" pod "407464c4-7b3c-4c02-9bde-db3e9a0e97a7" (UID: "407464c4-7b3c-4c02-9bde-db3e9a0e97a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:37.675159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.675096 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-token\" (UniqueName: \"kubernetes.io/projected/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-token\") pod \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " Oct 30 19:35:37.675275 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.675214 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-workload-socket" pod "407464c4-7b3c-4c02-9bde-db3e9a0e97a7" (UID: "407464c4-7b3c-4c02-9bde-db3e9a0e97a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:37.675315 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.675290 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-data" pod "407464c4-7b3c-4c02-9bde-db3e9a0e97a7" (UID: "407464c4-7b3c-4c02-9bde-db3e9a0e97a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:37.675357 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.675318 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-podinfo\") pod \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " Oct 30 19:35:37.675458 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.675437 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-workload-certs\") pod \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " Oct 30 19:35:37.675647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.675488 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istiod-ca-cert" pod "407464c4-7b3c-4c02-9bde-db3e9a0e97a7" (UID: "407464c4-7b3c-4c02-9bde-db3e9a0e97a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:37.675647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.675634 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-kube-api-access-8b9zs\" (UniqueName: \"kubernetes.io/projected/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-kube-api-access-8b9zs\") pod \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\" (UID: \"407464c4-7b3c-4c02-9bde-db3e9a0e97a7\") " Oct 30 19:35:37.676375 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.675698 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-workload-certs" pod "407464c4-7b3c-4c02-9bde-db3e9a0e97a7" (UID: "407464c4-7b3c-4c02-9bde-db3e9a0e97a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:37.676564 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.676539 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:37.676648 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.676572 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:37.676648 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.676587 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:37.676648 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.676602 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:37.676648 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.676618 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:37.677874 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.677845 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-podinfo" pod "407464c4-7b3c-4c02-9bde-db3e9a0e97a7" (UID: "407464c4-7b3c-4c02-9bde-db3e9a0e97a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:35:37.677959 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.677876 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-envoy" pod "407464c4-7b3c-4c02-9bde-db3e9a0e97a7" (UID: "407464c4-7b3c-4c02-9bde-db3e9a0e97a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:37.678054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.678000 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-kube-api-access-8b9zs" pod "407464c4-7b3c-4c02-9bde-db3e9a0e97a7" (UID: "407464c4-7b3c-4c02-9bde-db3e9a0e97a7"). InnerVolumeSpecName "kube-api-access-8b9zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:37.678353 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.678330 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-token" pod "407464c4-7b3c-4c02-9bde-db3e9a0e97a7" (UID: "407464c4-7b3c-4c02-9bde-db3e9a0e97a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:37.778717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.778677 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:37.778717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.778711 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b9zs\" (UniqueName: \"kubernetes.io/projected/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-kube-api-access-8b9zs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:37.778717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.778723 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:37.779035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:37.778736 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407464c4-7b3c-4c02-9bde-db3e9a0e97a7-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.024482 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.024442 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:35:38.115060 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.115024 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istiod-ca-cert\") pod \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " Oct 30 19:35:38.115266 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.115082 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-workload-certs\") pod \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " Oct 30 19:35:38.115266 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.115231 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-envoy\") pod \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " Oct 30 19:35:38.115384 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.115325 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-kube-api-access-b9sxf\" (UniqueName: \"kubernetes.io/projected/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-kube-api-access-b9sxf\") pod \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " Oct 30 19:35:38.115465 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.115438 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istiod-ca-cert" pod "d0e81adf-2dbf-41ff-a413-13ddc4310c6c" (UID: "d0e81adf-2dbf-41ff-a413-13ddc4310c6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:38.115547 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.115441 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-workload-certs" pod "d0e81adf-2dbf-41ff-a413-13ddc4310c6c" (UID: "d0e81adf-2dbf-41ff-a413-13ddc4310c6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:38.115547 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.115455 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-data\") pod \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " Oct 30 19:35:38.115715 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.115687 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-data" pod "d0e81adf-2dbf-41ff-a413-13ddc4310c6c" (UID: "d0e81adf-2dbf-41ff-a413-13ddc4310c6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:38.115715 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.115708 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-token\" (UniqueName: \"kubernetes.io/projected/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-token\") pod \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " Oct 30 19:35:38.115824 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.115762 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-podinfo\") pod \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " Oct 30 19:35:38.115824 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.115813 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-credential-socket\") pod \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " Oct 30 19:35:38.115932 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.115866 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-workload-socket\") pod \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\" (UID: \"d0e81adf-2dbf-41ff-a413-13ddc4310c6c\") " Oct 30 19:35:38.116573 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.116545 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-credential-socket" pod "d0e81adf-2dbf-41ff-a413-13ddc4310c6c" (UID: "d0e81adf-2dbf-41ff-a413-13ddc4310c6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:38.116687 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.116572 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.116687 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.116590 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.116687 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.116602 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.116854 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.116789 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-workload-socket" pod "d0e81adf-2dbf-41ff-a413-13ddc4310c6c" (UID: "d0e81adf-2dbf-41ff-a413-13ddc4310c6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:38.118193 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.118124 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-kube-api-access-b9sxf" pod "d0e81adf-2dbf-41ff-a413-13ddc4310c6c" (UID: "d0e81adf-2dbf-41ff-a413-13ddc4310c6c"). InnerVolumeSpecName "kube-api-access-b9sxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:38.118557 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.118533 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-envoy" pod "d0e81adf-2dbf-41ff-a413-13ddc4310c6c" (UID: "d0e81adf-2dbf-41ff-a413-13ddc4310c6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:38.119667 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.119645 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-token" pod "d0e81adf-2dbf-41ff-a413-13ddc4310c6c" (UID: "d0e81adf-2dbf-41ff-a413-13ddc4310c6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:38.120290 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.120265 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-podinfo" pod "d0e81adf-2dbf-41ff-a413-13ddc4310c6c" (UID: "d0e81adf-2dbf-41ff-a413-13ddc4310c6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:35:38.132647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.132613 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:35:38.218178 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.218140 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9399ee4d-b327-4739-85b1-d18eecea8116-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9399ee4d-b327-4739-85b1-d18eecea8116-istio-podinfo\") pod \"9399ee4d-b327-4739-85b1-d18eecea8116\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " Oct 30 19:35:38.218395 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.218207 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-workload-certs\") pod \"9399ee4d-b327-4739-85b1-d18eecea8116\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " Oct 30 19:35:38.218395 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.218259 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9399ee4d-b327-4739-85b1-d18eecea8116-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9399ee4d-b327-4739-85b1-d18eecea8116-istiod-ca-cert\") pod \"9399ee4d-b327-4739-85b1-d18eecea8116\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " Oct 30 19:35:38.218395 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.218331 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-istio-data\") pod \"9399ee4d-b327-4739-85b1-d18eecea8116\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " Oct 30 19:35:38.218395 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.218363 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-workload-socket\") pod \"9399ee4d-b327-4739-85b1-d18eecea8116\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " Oct 30 19:35:38.218632 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.218434 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-credential-socket\") pod \"9399ee4d-b327-4739-85b1-d18eecea8116\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " Oct 30 19:35:38.218632 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.218469 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9399ee4d-b327-4739-85b1-d18eecea8116-kube-api-access-fw7ld\" (UniqueName: \"kubernetes.io/projected/9399ee4d-b327-4739-85b1-d18eecea8116-kube-api-access-fw7ld\") pod \"9399ee4d-b327-4739-85b1-d18eecea8116\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " Oct 30 19:35:38.218632 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.218552 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-workload-certs" pod "9399ee4d-b327-4739-85b1-d18eecea8116" (UID: "9399ee4d-b327-4739-85b1-d18eecea8116"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:38.218632 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.218601 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9399ee4d-b327-4739-85b1-d18eecea8116-istio-token\" (UniqueName: \"kubernetes.io/projected/9399ee4d-b327-4739-85b1-d18eecea8116-istio-token\") pod \"9399ee4d-b327-4739-85b1-d18eecea8116\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " Oct 30 19:35:38.218904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.218639 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-istio-data" pod "9399ee4d-b327-4739-85b1-d18eecea8116" (UID: "9399ee4d-b327-4739-85b1-d18eecea8116"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:38.218904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.218701 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-istio-envoy\") pod \"9399ee4d-b327-4739-85b1-d18eecea8116\" (UID: \"9399ee4d-b327-4739-85b1-d18eecea8116\") " Oct 30 19:35:38.218904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.218762 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-credential-socket" pod "9399ee4d-b327-4739-85b1-d18eecea8116" (UID: "9399ee4d-b327-4739-85b1-d18eecea8116"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:38.218904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.218774 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9399ee4d-b327-4739-85b1-d18eecea8116-istiod-ca-cert" pod "9399ee4d-b327-4739-85b1-d18eecea8116" (UID: "9399ee4d-b327-4739-85b1-d18eecea8116"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:38.218904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.218900 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-workload-socket" pod "9399ee4d-b327-4739-85b1-d18eecea8116" (UID: "9399ee4d-b327-4739-85b1-d18eecea8116"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:38.219380 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.219354 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.219380 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.219379 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9399ee4d-b327-4739-85b1-d18eecea8116-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.219554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.219388 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.219554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.219396 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.219554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.219406 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.219554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.219414 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.219554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.219425 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9sxf\" (UniqueName: \"kubernetes.io/projected/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-kube-api-access-b9sxf\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.219554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.219434 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.219554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.219446 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.219554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.219458 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.219554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.219470 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e81adf-2dbf-41ff-a413-13ddc4310c6c-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.221562 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.221535 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-istio-envoy" pod "9399ee4d-b327-4739-85b1-d18eecea8116" (UID: "9399ee4d-b327-4739-85b1-d18eecea8116"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:38.221662 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.221575 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9399ee4d-b327-4739-85b1-d18eecea8116-kube-api-access-fw7ld" pod "9399ee4d-b327-4739-85b1-d18eecea8116" (UID: "9399ee4d-b327-4739-85b1-d18eecea8116"). InnerVolumeSpecName "kube-api-access-fw7ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:38.221662 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.221611 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9399ee4d-b327-4739-85b1-d18eecea8116-istio-token" pod "9399ee4d-b327-4739-85b1-d18eecea8116" (UID: "9399ee4d-b327-4739-85b1-d18eecea8116"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:38.221662 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.221650 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9399ee4d-b327-4739-85b1-d18eecea8116-istio-podinfo" pod "9399ee4d-b327-4739-85b1-d18eecea8116" (UID: "9399ee4d-b327-4739-85b1-d18eecea8116"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:35:38.321457 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.321359 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9399ee4d-b327-4739-85b1-d18eecea8116-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.321457 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.321399 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw7ld\" (UniqueName: \"kubernetes.io/projected/9399ee4d-b327-4739-85b1-d18eecea8116-kube-api-access-fw7ld\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.321457 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.321411 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9399ee4d-b327-4739-85b1-d18eecea8116-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.321457 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.321420 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9399ee4d-b327-4739-85b1-d18eecea8116-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:38.353534 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.353494 5047 generic.go:356] "Generic (PLEG): container finished" podID="407464c4-7b3c-4c02-9bde-db3e9a0e97a7" containerID="339aaa9d151381de549fbc0d1204d9c8fab4c0de6a8ce7c8692c5223eb0db804" exitCode=0 Oct 30 19:35:38.353729 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.353544 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" Oct 30 19:35:38.353729 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.353577 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" event={"ID":"407464c4-7b3c-4c02-9bde-db3e9a0e97a7","Type":"ContainerDied","Data":"339aaa9d151381de549fbc0d1204d9c8fab4c0de6a8ce7c8692c5223eb0db804"} Oct 30 19:35:38.353729 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.353637 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt" event={"ID":"407464c4-7b3c-4c02-9bde-db3e9a0e97a7","Type":"ContainerDied","Data":"d232ca23ab2b1ecd9be20acdc7afdd86523afb0ab9ff5df3f1ccf6668e77d66e"} Oct 30 19:35:38.353729 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.353664 5047 scope.go:122] "RemoveContainer" containerID="339aaa9d151381de549fbc0d1204d9c8fab4c0de6a8ce7c8692c5223eb0db804" Oct 30 19:35:38.355547 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.355521 5047 generic.go:356] "Generic (PLEG): container finished" podID="9399ee4d-b327-4739-85b1-d18eecea8116" containerID="5437db989a834dc6349d6ce5a9c5cf7ba10752132fa498067855c29f51b0380b" exitCode=0 Oct 30 19:35:38.355670 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.355587 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" Oct 30 19:35:38.355670 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.355621 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" event={"ID":"9399ee4d-b327-4739-85b1-d18eecea8116","Type":"ContainerDied","Data":"5437db989a834dc6349d6ce5a9c5cf7ba10752132fa498067855c29f51b0380b"} Oct 30 19:35:38.355807 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.355702 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj" event={"ID":"9399ee4d-b327-4739-85b1-d18eecea8116","Type":"ContainerDied","Data":"23b398febb3b415ce230df545bd77c8fd3e00dbbff38cef5ccc657999be64bee"} Oct 30 19:35:38.357740 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.357714 5047 generic.go:356] "Generic (PLEG): container finished" podID="d0e81adf-2dbf-41ff-a413-13ddc4310c6c" containerID="66548573291e74450d87f4fded492f635601cf85b59058f1672766cc7948a7fc" exitCode=0 Oct 30 19:35:38.357872 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.357763 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" event={"ID":"d0e81adf-2dbf-41ff-a413-13ddc4310c6c","Type":"ContainerDied","Data":"66548573291e74450d87f4fded492f635601cf85b59058f1672766cc7948a7fc"} Oct 30 19:35:38.357872 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.357790 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" Oct 30 19:35:38.357872 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.357805 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c" event={"ID":"d0e81adf-2dbf-41ff-a413-13ddc4310c6c","Type":"ContainerDied","Data":"e5d58ea678fac790ff19b923695b4bb92294608803b2ba52e55864bcb8597763"} Oct 30 19:35:38.369550 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.369527 5047 scope.go:122] "RemoveContainer" containerID="339aaa9d151381de549fbc0d1204d9c8fab4c0de6a8ce7c8692c5223eb0db804" Oct 30 19:35:38.369976 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:35:38.369911 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339aaa9d151381de549fbc0d1204d9c8fab4c0de6a8ce7c8692c5223eb0db804\": container with ID starting with 339aaa9d151381de549fbc0d1204d9c8fab4c0de6a8ce7c8692c5223eb0db804 not found: ID does not exist" containerID="339aaa9d151381de549fbc0d1204d9c8fab4c0de6a8ce7c8692c5223eb0db804" Oct 30 19:35:38.369976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.369949 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339aaa9d151381de549fbc0d1204d9c8fab4c0de6a8ce7c8692c5223eb0db804"} err="failed to get container status \"339aaa9d151381de549fbc0d1204d9c8fab4c0de6a8ce7c8692c5223eb0db804\": rpc error: code = NotFound desc = could not find container \"339aaa9d151381de549fbc0d1204d9c8fab4c0de6a8ce7c8692c5223eb0db804\": container with ID starting with 339aaa9d151381de549fbc0d1204d9c8fab4c0de6a8ce7c8692c5223eb0db804 not found: ID does not exist" Oct 30 19:35:38.369976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.369974 5047 scope.go:122] "RemoveContainer" containerID="5437db989a834dc6349d6ce5a9c5cf7ba10752132fa498067855c29f51b0380b" Oct 30 19:35:38.382082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.382060 5047 scope.go:122] "RemoveContainer" containerID="5437db989a834dc6349d6ce5a9c5cf7ba10752132fa498067855c29f51b0380b" Oct 30 19:35:38.382449 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:35:38.382403 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5437db989a834dc6349d6ce5a9c5cf7ba10752132fa498067855c29f51b0380b\": container with ID starting with 5437db989a834dc6349d6ce5a9c5cf7ba10752132fa498067855c29f51b0380b not found: ID does not exist" containerID="5437db989a834dc6349d6ce5a9c5cf7ba10752132fa498067855c29f51b0380b" Oct 30 19:35:38.382538 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.382453 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5437db989a834dc6349d6ce5a9c5cf7ba10752132fa498067855c29f51b0380b"} err="failed to get container status \"5437db989a834dc6349d6ce5a9c5cf7ba10752132fa498067855c29f51b0380b\": rpc error: code = NotFound desc = could not find container \"5437db989a834dc6349d6ce5a9c5cf7ba10752132fa498067855c29f51b0380b\": container with ID starting with 5437db989a834dc6349d6ce5a9c5cf7ba10752132fa498067855c29f51b0380b not found: ID does not exist" Oct 30 19:35:38.382538 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.382474 5047 scope.go:122] "RemoveContainer" containerID="66548573291e74450d87f4fded492f635601cf85b59058f1672766cc7948a7fc" Oct 30 19:35:38.393232 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.393210 5047 scope.go:122] "RemoveContainer" containerID="66548573291e74450d87f4fded492f635601cf85b59058f1672766cc7948a7fc" Oct 30 19:35:38.393592 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:35:38.393568 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66548573291e74450d87f4fded492f635601cf85b59058f1672766cc7948a7fc\": container with ID starting with 66548573291e74450d87f4fded492f635601cf85b59058f1672766cc7948a7fc not found: ID does not exist" containerID="66548573291e74450d87f4fded492f635601cf85b59058f1672766cc7948a7fc" Oct 30 19:35:38.393637 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.393603 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66548573291e74450d87f4fded492f635601cf85b59058f1672766cc7948a7fc"} err="failed to get container status \"66548573291e74450d87f4fded492f635601cf85b59058f1672766cc7948a7fc\": rpc error: code = NotFound desc = could not find container \"66548573291e74450d87f4fded492f635601cf85b59058f1672766cc7948a7fc\": container with ID starting with 66548573291e74450d87f4fded492f635601cf85b59058f1672766cc7948a7fc not found: ID does not exist" Oct 30 19:35:38.433108 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.433078 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj"] Oct 30 19:35:38.437340 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.437314 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-2nslj"] Oct 30 19:35:38.469690 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.469639 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c"] Oct 30 19:35:38.473663 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.473637 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-f6v8c"] Oct 30 19:35:38.504241 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.504200 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt"] Oct 30 19:35:38.508299 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:38.508271 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-g5pdt"] Oct 30 19:35:39.053362 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:39.053324 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407464c4-7b3c-4c02-9bde-db3e9a0e97a7" path="/var/lib/kubelet/pods/407464c4-7b3c-4c02-9bde-db3e9a0e97a7/volumes" Oct 30 19:35:39.054146 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:39.054125 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9399ee4d-b327-4739-85b1-d18eecea8116" path="/var/lib/kubelet/pods/9399ee4d-b327-4739-85b1-d18eecea8116/volumes" Oct 30 19:35:39.054810 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:39.054793 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0e81adf-2dbf-41ff-a413-13ddc4310c6c" path="/var/lib/kubelet/pods/d0e81adf-2dbf-41ff-a413-13ddc4310c6c/volumes" Oct 30 19:35:39.614167 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:35:39.614118 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:35:48.378169 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.378128 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd"] Oct 30 19:35:48.439045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.379832 5047 cpu_manager.go:414] "containerMap: removing container" podUID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" containerName="istio-proxy" Oct 30 19:35:48.568999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.566506 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" containerName="istio-proxy" Oct 30 19:35:48.568999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.566549 5047 cpu_manager.go:414] "containerMap: removing container" podUID="407464c4-7b3c-4c02-9bde-db3e9a0e97a7" containerName="istio-proxy" Oct 30 19:35:48.568999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.566560 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407464c4-7b3c-4c02-9bde-db3e9a0e97a7" containerName="istio-proxy" Oct 30 19:35:48.568999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.566595 5047 cpu_manager.go:414] "containerMap: removing container" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" containerName="istio-proxy" Oct 30 19:35:48.568999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.566604 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" containerName="istio-proxy" Oct 30 19:35:48.568999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.566634 5047 cpu_manager.go:414] "containerMap: removing container" podUID="820cab28-9ecc-46d5-945a-df0b07cf3c4f" containerName="payload-pre-processing" Oct 30 19:35:48.568999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.566642 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820cab28-9ecc-46d5-945a-df0b07cf3c4f" containerName="payload-pre-processing" Oct 30 19:35:48.568999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.566676 5047 cpu_manager.go:414] "containerMap: removing container" podUID="9399ee4d-b327-4739-85b1-d18eecea8116" containerName="istio-proxy" Oct 30 19:35:48.568999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.566684 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9399ee4d-b327-4739-85b1-d18eecea8116" containerName="istio-proxy" Oct 30 19:35:48.568999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.566702 5047 cpu_manager.go:414] "containerMap: removing container" podUID="d0e81adf-2dbf-41ff-a413-13ddc4310c6c" containerName="istio-proxy" Oct 30 19:35:48.568999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.566709 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e81adf-2dbf-41ff-a413-13ddc4310c6c" containerName="istio-proxy" Oct 30 19:35:48.568999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.567064 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="9399ee4d-b327-4739-85b1-d18eecea8116" containerName="istio-proxy" Oct 30 19:35:48.568999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.567089 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e81adf-2dbf-41ff-a413-13ddc4310c6c" containerName="istio-proxy" Oct 30 19:35:48.568999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.567118 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c276d2-2b71-4701-a4f2-bf3dbc9f9a75" containerName="istio-proxy" Oct 30 19:35:48.568999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.567127 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="407464c4-7b3c-4c02-9bde-db3e9a0e97a7" containerName="istio-proxy" Oct 30 19:35:48.568999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.567139 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="c080477b-5afb-4972-8feb-1bd6ec66092f" containerName="istio-proxy" Oct 30 19:35:48.568999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.567151 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="820cab28-9ecc-46d5-945a-df0b07cf3c4f" containerName="payload-pre-processing" Oct 30 19:35:48.765119 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.747489 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd"] Oct 30 19:35:48.773137 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.751184 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:48.831125 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.829288 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d4cb13-openshift-default-dockercfg-tt67b\"" Oct 30 19:35:48.973423 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:48.973377 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkjxp\" (UniqueName: \"kubernetes.io/projected/90f0e672-c84b-4721-9a09-363ffdf05947-kube-api-access-mkjxp\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.001676 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.001631 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-workload-socket\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.001975 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.001952 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f0e672-c84b-4721-9a09-363ffdf05947-istiod-ca-cert\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.002441 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.002419 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f0e672-c84b-4721-9a09-363ffdf05947-istio-podinfo\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.002612 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.002592 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-credential-socket\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.002815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.002798 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-istio-envoy\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.002982 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.002966 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-workload-certs\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.003188 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.003170 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f0e672-c84b-4721-9a09-363ffdf05947-istio-token\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.003388 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.003369 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-istio-data\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.105739 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.105319 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkjxp\" (UniqueName: \"kubernetes.io/projected/90f0e672-c84b-4721-9a09-363ffdf05947-kube-api-access-mkjxp\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.108179 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.108140 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-workload-socket\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.108344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.108199 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f0e672-c84b-4721-9a09-363ffdf05947-istiod-ca-cert\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.108344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.108251 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f0e672-c84b-4721-9a09-363ffdf05947-istio-podinfo\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.108344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.108287 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-credential-socket\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.108524 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.108381 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-istio-envoy\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.108524 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.108469 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-workload-certs\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.108623 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.108552 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f0e672-c84b-4721-9a09-363ffdf05947-istio-token\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.108679 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.108649 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-istio-data\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.109223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.109195 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-workload-socket\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.109515 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.109473 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f0e672-c84b-4721-9a09-363ffdf05947-istiod-ca-cert\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.109703 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.109506 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-workload-certs\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.109846 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.109615 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-istio-data\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.109949 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.109741 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-credential-socket\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.111608 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.111584 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-istio-envoy\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.112837 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.112791 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f0e672-c84b-4721-9a09-363ffdf05947-istio-podinfo\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.126194 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.126167 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkjxp\" (UniqueName: \"kubernetes.io/projected/90f0e672-c84b-4721-9a09-363ffdf05947-kube-api-access-mkjxp\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.129076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.129048 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f0e672-c84b-4721-9a09-363ffdf05947-istio-token\") pod \"e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.254030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.247384 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:49.793166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.793134 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd"] Oct 30 19:35:49.793509 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:35:49.793188 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f0e672_c84b_4721_9a09_363ffdf05947.slice/crio-80ee50a156b2daf9e45d1c61e55a1f981deb9ea137474fa153bd70bd0eb7493c WatchSource:0}: Error finding container 80ee50a156b2daf9e45d1c61e55a1f981deb9ea137474fa153bd70bd0eb7493c: Status 404 returned error can't find the container with id 80ee50a156b2daf9e45d1c61e55a1f981deb9ea137474fa153bd70bd0eb7493c Oct 30 19:35:49.796239 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.796206 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:49.796421 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.796288 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:49.796421 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.796334 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:49.910085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:49.909719 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" event={"ID":"90f0e672-c84b-4721-9a09-363ffdf05947","Type":"ContainerStarted","Data":"80ee50a156b2daf9e45d1c61e55a1f981deb9ea137474fa153bd70bd0eb7493c"} Oct 30 19:35:50.920171 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:50.920126 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" event={"ID":"90f0e672-c84b-4721-9a09-363ffdf05947","Type":"ContainerStarted","Data":"b1250f61e90a56e35aa35cdae53794312ff17d973da85d4631f4b2a7abad7105"} Oct 30 19:35:50.999656 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:50.999594 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" podStartSLOduration=2.9995798970000003 podStartE2EDuration="2.999579897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:50.986636381 +0000 UTC m=+1904.061091776" watchObservedRunningTime="2026-10-30 19:35:50.999579897 +0000 UTC m=+1904.074035255" Oct 30 19:35:51.256205 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:51.256081 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:51.298910 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:51.298876 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:51.932369 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:51.932331 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:51.933539 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:51.933514 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:35:55.381918 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.381878 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25"] Oct 30 19:35:55.389132 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.389094 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" Oct 30 19:35:55.410334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.410288 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25"] Oct 30 19:35:55.416243 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.416209 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d4cb13\"" Oct 30 19:35:55.476048 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.443439 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d4cb13-dockercfg-65bmb\"" Oct 30 19:35:55.556036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.554550 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh9gp\" (UniqueName: \"kubernetes.io/projected/d86a88d1-aeee-4dd6-bbc3-a627fdd4854c-kube-api-access-kh9gp\") pod \"payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25\" (UID: \"d86a88d1-aeee-4dd6-bbc3-a627fdd4854c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" Oct 30 19:35:55.556036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.555061 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86a88d1-aeee-4dd6-bbc3-a627fdd4854c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25\" (UID: \"d86a88d1-aeee-4dd6-bbc3-a627fdd4854c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" Oct 30 19:35:55.717800 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.716340 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86a88d1-aeee-4dd6-bbc3-a627fdd4854c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25\" (UID: \"d86a88d1-aeee-4dd6-bbc3-a627fdd4854c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" Oct 30 19:35:55.717800 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.716471 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh9gp\" (UniqueName: \"kubernetes.io/projected/d86a88d1-aeee-4dd6-bbc3-a627fdd4854c-kube-api-access-kh9gp\") pod \"payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25\" (UID: \"d86a88d1-aeee-4dd6-bbc3-a627fdd4854c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" Oct 30 19:35:55.718716 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.718476 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86a88d1-aeee-4dd6-bbc3-a627fdd4854c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25\" (UID: \"d86a88d1-aeee-4dd6-bbc3-a627fdd4854c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" Oct 30 19:35:55.795072 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.791779 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh9gp\" (UniqueName: \"kubernetes.io/projected/d86a88d1-aeee-4dd6-bbc3-a627fdd4854c-kube-api-access-kh9gp\") pod \"payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25\" (UID: \"d86a88d1-aeee-4dd6-bbc3-a627fdd4854c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" Oct 30 19:35:55.795942 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.795906 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk"] Oct 30 19:35:55.823023 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.822306 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk"] Oct 30 19:35:55.823967 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.823943 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:55.924598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.924554 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:55.924835 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.924807 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:55.924896 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.924845 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:55.924994 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.924977 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:55.925087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.925066 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:55.925198 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.925183 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:55.925250 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.925222 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:55.925360 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.925336 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz59s\" (UniqueName: \"kubernetes.io/projected/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-kube-api-access-wz59s\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:55.925429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:55.925415 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.009942 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.009843 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" Oct 30 19:35:56.032523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.031211 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.032523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.031277 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.032523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.031367 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.032523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.031420 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.032523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.031524 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.032523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.031566 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.032523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.031604 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz59s\" (UniqueName: \"kubernetes.io/projected/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-kube-api-access-wz59s\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.032523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.031695 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.032523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.031767 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.032523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.031970 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.033314 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.032723 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.033314 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.032997 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.033518 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.033488 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.036502 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.036468 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.038551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.038506 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.047343 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.047270 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.053923 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.053890 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz59s\" (UniqueName: \"kubernetes.io/projected/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-kube-api-access-wz59s\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.055468 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.055435 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.152601 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.152434 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:56.240416 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.233409 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5"] Oct 30 19:35:56.240416 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.233676 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" podUID="ccac5abe-13ee-4832-9b5a-38029145f82c" containerName="maas-api" containerID="cri-o://21049643a74e9fdc6f92dcc606343b4421f1c4bb721d9d28505d81a97ca00235" gracePeriod=30 Oct 30 19:35:56.350655 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.349043 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25"] Oct 30 19:35:56.673316 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:35:56.673268 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908307f7_c52b_4bdc_b5f9_5d7ea7ea57eb.slice/crio-a6bdd32d42472e1eb1ec320cc76e2787af416b0114ca23d1f636878c819128e3 WatchSource:0}: Error finding container a6bdd32d42472e1eb1ec320cc76e2787af416b0114ca23d1f636878c819128e3: Status 404 returned error can't find the container with id a6bdd32d42472e1eb1ec320cc76e2787af416b0114ca23d1f636878c819128e3 Oct 30 19:35:56.695214 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.677210 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:56.695214 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.677304 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:56.695214 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.677347 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:56.865209 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:56.849837 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk"] Oct 30 19:35:57.149555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.149298 5047 generic.go:356] "Generic (PLEG): container finished" podID="ccac5abe-13ee-4832-9b5a-38029145f82c" containerID="21049643a74e9fdc6f92dcc606343b4421f1c4bb721d9d28505d81a97ca00235" exitCode=0 Oct 30 19:35:57.192441 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.187262 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" event={"ID":"d86a88d1-aeee-4dd6-bbc3-a627fdd4854c","Type":"ContainerStarted","Data":"3826a3cbb86d2ce0a39335345f6607e2237ce6469e594dcdd1cf3621f55a1cc0"} Oct 30 19:35:57.192441 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.187347 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" event={"ID":"d86a88d1-aeee-4dd6-bbc3-a627fdd4854c","Type":"ContainerStarted","Data":"ab453cd8a0fd1d54dffc74bbbd0d4ee1d696a1fbbc67384eeaaa19fb57d9dae5"} Oct 30 19:35:57.192441 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.187381 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" event={"ID":"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb","Type":"ContainerStarted","Data":"a6bdd32d42472e1eb1ec320cc76e2787af416b0114ca23d1f636878c819128e3"} Oct 30 19:35:57.192441 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.187417 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" event={"ID":"ccac5abe-13ee-4832-9b5a-38029145f82c","Type":"ContainerDied","Data":"21049643a74e9fdc6f92dcc606343b4421f1c4bb721d9d28505d81a97ca00235"} Oct 30 19:35:57.229144 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.228743 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:35:57.232438 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.232371 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" podStartSLOduration=2.232351157 podStartE2EDuration="2.232351157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:57.228183239 +0000 UTC m=+1910.302638599" watchObservedRunningTime="2026-10-30 19:35:57.232351157 +0000 UTC m=+1910.306806519" Oct 30 19:35:57.297095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.295538 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccac5abe-13ee-4832-9b5a-38029145f82c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccac5abe-13ee-4832-9b5a-38029145f82c-metrics-certs\") pod \"ccac5abe-13ee-4832-9b5a-38029145f82c\" (UID: \"ccac5abe-13ee-4832-9b5a-38029145f82c\") " Oct 30 19:35:57.297095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.295656 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccac5abe-13ee-4832-9b5a-38029145f82c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccac5abe-13ee-4832-9b5a-38029145f82c-openshift-service-ca\") pod \"ccac5abe-13ee-4832-9b5a-38029145f82c\" (UID: \"ccac5abe-13ee-4832-9b5a-38029145f82c\") " Oct 30 19:35:57.297095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.295893 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccac5abe-13ee-4832-9b5a-38029145f82c-kube-api-access-8lvrg\" (UniqueName: \"kubernetes.io/projected/ccac5abe-13ee-4832-9b5a-38029145f82c-kube-api-access-8lvrg\") pod \"ccac5abe-13ee-4832-9b5a-38029145f82c\" (UID: \"ccac5abe-13ee-4832-9b5a-38029145f82c\") " Oct 30 19:35:57.297095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.296025 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccac5abe-13ee-4832-9b5a-38029145f82c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccac5abe-13ee-4832-9b5a-38029145f82c-maas-api-tls\") pod \"ccac5abe-13ee-4832-9b5a-38029145f82c\" (UID: \"ccac5abe-13ee-4832-9b5a-38029145f82c\") " Oct 30 19:35:57.298514 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.298210 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccac5abe-13ee-4832-9b5a-38029145f82c-openshift-service-ca" pod "ccac5abe-13ee-4832-9b5a-38029145f82c" (UID: "ccac5abe-13ee-4832-9b5a-38029145f82c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:57.302433 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.302394 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccac5abe-13ee-4832-9b5a-38029145f82c-metrics-certs" pod "ccac5abe-13ee-4832-9b5a-38029145f82c" (UID: "ccac5abe-13ee-4832-9b5a-38029145f82c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:57.314066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.312267 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccac5abe-13ee-4832-9b5a-38029145f82c-maas-api-tls" pod "ccac5abe-13ee-4832-9b5a-38029145f82c" (UID: "ccac5abe-13ee-4832-9b5a-38029145f82c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:57.314066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.312444 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccac5abe-13ee-4832-9b5a-38029145f82c-kube-api-access-8lvrg" pod "ccac5abe-13ee-4832-9b5a-38029145f82c" (UID: "ccac5abe-13ee-4832-9b5a-38029145f82c"). InnerVolumeSpecName "kube-api-access-8lvrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:57.447804 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.431292 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lvrg\" (UniqueName: \"kubernetes.io/projected/ccac5abe-13ee-4832-9b5a-38029145f82c-kube-api-access-8lvrg\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:57.447804 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.431337 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccac5abe-13ee-4832-9b5a-38029145f82c-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:57.447804 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.431352 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccac5abe-13ee-4832-9b5a-38029145f82c-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:57.447804 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:57.431365 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccac5abe-13ee-4832-9b5a-38029145f82c-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:35:58.179468 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:58.179423 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" event={"ID":"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb","Type":"ContainerStarted","Data":"0d81bdb27dc6388fc7151f31d572e32217f14a8614b8f06b880b4a6ce622bf54"} Oct 30 19:35:58.181251 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:58.181226 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" Oct 30 19:35:58.181393 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:58.181258 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5" event={"ID":"ccac5abe-13ee-4832-9b5a-38029145f82c","Type":"ContainerDied","Data":"81fde51a6aa145c2fa98312574d4ae1d460871ecfaa80cafc304c40e2752e68d"} Oct 30 19:35:58.181393 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:58.181321 5047 scope.go:122] "RemoveContainer" containerID="21049643a74e9fdc6f92dcc606343b4421f1c4bb721d9d28505d81a97ca00235" Oct 30 19:35:58.181577 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:58.181555 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" Oct 30 19:35:58.273459 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:58.273406 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" podStartSLOduration=3.273390565 podStartE2EDuration="3.273390565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:58.233349472 +0000 UTC m=+1911.307804832" watchObservedRunningTime="2026-10-30 19:35:58.273390565 +0000 UTC m=+1911.347845991" Oct 30 19:35:58.295163 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:58.295127 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5"] Oct 30 19:35:58.299064 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:58.299039 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ccf8bd61-98d77cbcd-8xbj5"] Oct 30 19:35:59.077213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:59.077167 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccac5abe-13ee-4832-9b5a-38029145f82c" path="/var/lib/kubelet/pods/ccac5abe-13ee-4832-9b5a-38029145f82c/volumes" Oct 30 19:35:59.154184 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:59.154071 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:35:59.160997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:59.160508 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:35:59.160997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:59.160662 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" podUID="908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 19:35:59.193108 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:59.193078 5047 generic.go:356] "Generic (PLEG): container finished" podID="c10e3c6c-bbf1-4606-990e-bc2898aff682" containerID="29a21cb49fc2a3a1f5364edbc0955a65f06c05ab088daede9233aefc759f059f" exitCode=0 Oct 30 19:35:59.193615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:59.193151 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" event={"ID":"c10e3c6c-bbf1-4606-990e-bc2898aff682","Type":"ContainerDied","Data":"29a21cb49fc2a3a1f5364edbc0955a65f06c05ab088daede9233aefc759f059f"} Oct 30 19:35:59.193950 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:59.193930 5047 scope.go:122] "RemoveContainer" containerID="29a21cb49fc2a3a1f5364edbc0955a65f06c05ab088daede9233aefc759f059f" Oct 30 19:35:59.195825 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:59.195797 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:59.195936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:59.195870 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:59.195936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:59.195911 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:59.230164 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:59.229816 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": read tcp 10.132.0.2:53030->10.132.0.123:15021: read: connection reset by peer" start-of-body= Oct 30 19:35:59.230164 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:35:59.229892 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" podUID="a252f556-2d81-4164-8efa-65037946f382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": read tcp 10.132.0.2:53030->10.132.0.123:15021: read: connection reset by peer" Oct 30 19:36:00.159473 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:00.159426 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:36:00.159686 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:00.159496 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" podUID="908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 19:36:00.203171 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:00.203140 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" event={"ID":"c10e3c6c-bbf1-4606-990e-bc2898aff682","Type":"ContainerStarted","Data":"446d0473ff1c0841ad2cf8676751bc7fa653ec3f6f22515dffffb4620ae987f5"} Oct 30 19:36:00.207136 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:00.207110 5047 generic.go:356] "Generic (PLEG): container finished" podID="a252f556-2d81-4164-8efa-65037946f382" containerID="a1348b944313cd6a43577eb3d495ef73a1c9e16ed41a301461d8fa11773d4559" exitCode=0 Oct 30 19:36:00.207376 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:00.207176 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" event={"ID":"a252f556-2d81-4164-8efa-65037946f382","Type":"ContainerDied","Data":"a1348b944313cd6a43577eb3d495ef73a1c9e16ed41a301461d8fa11773d4559"} Oct 30 19:36:00.207773 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:00.207754 5047 scope.go:122] "RemoveContainer" containerID="a1348b944313cd6a43577eb3d495ef73a1c9e16ed41a301461d8fa11773d4559" Oct 30 19:36:00.209480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:00.209449 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:00.209560 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:00.209524 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:00.209600 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:00.209562 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:00.759028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:00.758459 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:36:00.759028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:00.758606 5047 patch_prober.go:29] interesting pod/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 19:36:00.759028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:00.758649 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" podUID="c10e3c6c-bbf1-4606-990e-bc2898aff682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 19:36:01.180077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:01.179757 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:36:01.180077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:01.179825 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" podUID="908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 19:36:01.528344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:01.528238 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" event={"ID":"a252f556-2d81-4164-8efa-65037946f382","Type":"ContainerStarted","Data":"1283e93829ab77a30c4b384bddfd704c8bc6c269937b20cc979c006cb3bd82bd"} Oct 30 19:36:01.757704 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:01.757224 5047 patch_prober.go:29] interesting pod/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 19:36:01.757704 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:01.757288 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" podUID="c10e3c6c-bbf1-4606-990e-bc2898aff682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 19:36:02.160031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:02.159495 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:36:02.160031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:02.159559 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" podUID="908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 19:36:02.282030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:02.279415 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:36:02.282030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:02.281115 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 19:36:02.282030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:02.281171 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" podUID="a252f556-2d81-4164-8efa-65037946f382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 19:36:02.777756 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:02.777258 5047 patch_prober.go:29] interesting pod/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 19:36:02.777756 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:02.777324 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" podUID="c10e3c6c-bbf1-4606-990e-bc2898aff682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 19:36:03.162043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:03.160528 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:36:03.162043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:03.160603 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" podUID="908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 19:36:03.283330 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:03.283290 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 19:36:03.283615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:03.283586 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" podUID="a252f556-2d81-4164-8efa-65037946f382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 19:36:03.775195 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:03.775149 5047 patch_prober.go:29] interesting pod/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 19:36:03.775475 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:03.775438 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" podUID="c10e3c6c-bbf1-4606-990e-bc2898aff682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 19:36:04.016058 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:04.016020 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:36:04.156424 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:04.156365 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:36:04.156659 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:04.156446 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" podUID="908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 19:36:04.282027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:04.281396 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 19:36:04.282027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:04.281482 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" podUID="a252f556-2d81-4164-8efa-65037946f382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 19:36:04.759030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:04.758489 5047 patch_prober.go:29] interesting pod/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 19:36:04.759030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:04.758581 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" podUID="c10e3c6c-bbf1-4606-990e-bc2898aff682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 19:36:05.185032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:05.184402 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:36:05.187078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:05.186270 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:36:05.188165 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:05.188133 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:36:05.290033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:05.289488 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 19:36:05.290033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:05.289586 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" podUID="a252f556-2d81-4164-8efa-65037946f382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 19:36:05.771581 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:05.771533 5047 patch_prober.go:29] interesting pod/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 19:36:05.771945 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:05.771912 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" podUID="c10e3c6c-bbf1-4606-990e-bc2898aff682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 19:36:06.089443 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.089403 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98"] Oct 30 19:36:06.090284 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.090263 5047 cpu_manager.go:414] "containerMap: removing container" podUID="ccac5abe-13ee-4832-9b5a-38029145f82c" containerName="maas-api" Oct 30 19:36:06.090361 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.090285 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccac5abe-13ee-4832-9b5a-38029145f82c" containerName="maas-api" Oct 30 19:36:06.090463 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.090453 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccac5abe-13ee-4832-9b5a-38029145f82c" containerName="maas-api" Oct 30 19:36:06.094795 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.094763 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" Oct 30 19:36:06.108861 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.108831 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ccf8bd61\"" Oct 30 19:36:06.109246 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.109222 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ccf8bd61-dockercfg-dsht9\"" Oct 30 19:36:06.115823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.115779 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98"] Oct 30 19:36:06.229072 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.226198 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9deac3a-adad-4f75-8c98-d4fdeeb5b223-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98\" (UID: \"b9deac3a-adad-4f75-8c98-d4fdeeb5b223\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" Oct 30 19:36:06.229072 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.226472 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kvsq\" (UniqueName: \"kubernetes.io/projected/b9deac3a-adad-4f75-8c98-d4fdeeb5b223-kube-api-access-6kvsq\") pod \"payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98\" (UID: \"b9deac3a-adad-4f75-8c98-d4fdeeb5b223\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" Oct 30 19:36:06.285036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.231575 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx"] Oct 30 19:36:06.345085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.283486 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 19:36:06.345085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.290111 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" podUID="a252f556-2d81-4164-8efa-65037946f382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 19:36:06.388032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.348098 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kvsq\" (UniqueName: \"kubernetes.io/projected/b9deac3a-adad-4f75-8c98-d4fdeeb5b223-kube-api-access-6kvsq\") pod \"payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98\" (UID: \"b9deac3a-adad-4f75-8c98-d4fdeeb5b223\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" Oct 30 19:36:06.388032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.348315 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9deac3a-adad-4f75-8c98-d4fdeeb5b223-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98\" (UID: \"b9deac3a-adad-4f75-8c98-d4fdeeb5b223\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" Oct 30 19:36:06.413038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.389126 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9deac3a-adad-4f75-8c98-d4fdeeb5b223-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98\" (UID: \"b9deac3a-adad-4f75-8c98-d4fdeeb5b223\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" Oct 30 19:36:06.613122 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.597605 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx"] Oct 30 19:36:06.613122 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.597755 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" Oct 30 19:36:06.658070 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.652757 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kvsq\" (UniqueName: \"kubernetes.io/projected/b9deac3a-adad-4f75-8c98-d4fdeeb5b223-kube-api-access-6kvsq\") pod \"payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98\" (UID: \"b9deac3a-adad-4f75-8c98-d4fdeeb5b223\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" Oct 30 19:36:06.696062 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.696020 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc1de489-6ed4-4299-9af0-e22c3f80dbb3-plugins-config-volume\") pod \"payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx\" (UID: \"cc1de489-6ed4-4299-9af0-e22c3f80dbb3\") " pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" Oct 30 19:36:06.696489 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.696469 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvkpt\" (UniqueName: \"kubernetes.io/projected/cc1de489-6ed4-4299-9af0-e22c3f80dbb3-kube-api-access-hvkpt\") pod \"payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx\" (UID: \"cc1de489-6ed4-4299-9af0-e22c3f80dbb3\") " pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" Oct 30 19:36:06.712173 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.712134 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" Oct 30 19:36:06.771958 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.771931 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:36:06.773500 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.773479 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:36:06.799059 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.798458 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvkpt\" (UniqueName: \"kubernetes.io/projected/cc1de489-6ed4-4299-9af0-e22c3f80dbb3-kube-api-access-hvkpt\") pod \"payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx\" (UID: \"cc1de489-6ed4-4299-9af0-e22c3f80dbb3\") " pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" Oct 30 19:36:06.799059 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.798685 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc1de489-6ed4-4299-9af0-e22c3f80dbb3-plugins-config-volume\") pod \"payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx\" (UID: \"cc1de489-6ed4-4299-9af0-e22c3f80dbb3\") " pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" Oct 30 19:36:06.800089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.799994 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc1de489-6ed4-4299-9af0-e22c3f80dbb3-plugins-config-volume\") pod \"payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx\" (UID: \"cc1de489-6ed4-4299-9af0-e22c3f80dbb3\") " pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" Oct 30 19:36:06.829035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.828970 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvkpt\" (UniqueName: \"kubernetes.io/projected/cc1de489-6ed4-4299-9af0-e22c3f80dbb3-kube-api-access-hvkpt\") pod \"payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx\" (UID: \"cc1de489-6ed4-4299-9af0-e22c3f80dbb3\") " pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" Oct 30 19:36:06.951283 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:06.948765 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" Oct 30 19:36:07.111739 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:07.111706 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98"] Oct 30 19:36:07.279726 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:07.279696 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 19:36:07.280132 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:07.279747 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" podUID="a252f556-2d81-4164-8efa-65037946f382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 19:36:07.309114 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:36:07.309082 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1de489_6ed4_4299_9af0_e22c3f80dbb3.slice/crio-25bdb3859c39c44055e09fc059b61b7c8473d65d89f681e051519921904f9653 WatchSource:0}: Error finding container 25bdb3859c39c44055e09fc059b61b7c8473d65d89f681e051519921904f9653: Status 404 returned error can't find the container with id 25bdb3859c39c44055e09fc059b61b7c8473d65d89f681e051519921904f9653 Oct 30 19:36:07.310870 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:07.310837 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx"] Oct 30 19:36:07.778647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:07.778587 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" event={"ID":"b9deac3a-adad-4f75-8c98-d4fdeeb5b223","Type":"ContainerStarted","Data":"ffc94697db48c0bbf3a760bd230c5b41a9f0d3202f87308895904b01090c9cf0"} Oct 30 19:36:07.778873 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:07.778709 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" Oct 30 19:36:07.778873 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:07.778737 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" event={"ID":"b9deac3a-adad-4f75-8c98-d4fdeeb5b223","Type":"ContainerStarted","Data":"3ac5f0c7b68cf1f925d69a50c109b713ec4f6ee810d564511a8b118bf326a554"} Oct 30 19:36:07.780527 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:07.780485 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" event={"ID":"cc1de489-6ed4-4299-9af0-e22c3f80dbb3","Type":"ContainerStarted","Data":"28cb4b24386d03e39bb5a4af7885c6ffe0d57cd4cf07c07e40099b8b57c35243"} Oct 30 19:36:07.780672 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:07.780547 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" event={"ID":"cc1de489-6ed4-4299-9af0-e22c3f80dbb3","Type":"ContainerStarted","Data":"25bdb3859c39c44055e09fc059b61b7c8473d65d89f681e051519921904f9653"} Oct 30 19:36:07.780672 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:07.780568 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" Oct 30 19:36:07.822803 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:07.822740 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" podStartSLOduration=1.822726407 podStartE2EDuration="1.822726407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:07.818180656 +0000 UTC m=+1920.892636028" watchObservedRunningTime="2026-10-30 19:36:07.822726407 +0000 UTC m=+1920.897181766" Oct 30 19:36:07.859403 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:07.859338 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" podStartSLOduration=1.85931913 podStartE2EDuration="1.85931913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:07.855349077 +0000 UTC m=+1920.929804435" watchObservedRunningTime="2026-10-30 19:36:07.85931913 +0000 UTC m=+1920.933774513" Oct 30 19:36:08.283839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.283805 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:36:08.284357 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.284305 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:36:08.285036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.285000 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:36:08.317050 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.316984 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm"] Oct 30 19:36:08.320893 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.320868 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:36:08.343246 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.340900 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8e6b67-serving-cert\"" Oct 30 19:36:08.348083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.347064 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm"] Oct 30 19:36:08.443210 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.443168 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12e1654-8408-4422-8f28-37dbb4a77d49-metrics-certs\") pod \"maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm\" (UID: \"a12e1654-8408-4422-8f28-37dbb4a77d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:36:08.443427 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.443235 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86snq\" (UniqueName: \"kubernetes.io/projected/a12e1654-8408-4422-8f28-37dbb4a77d49-kube-api-access-86snq\") pod \"maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm\" (UID: \"a12e1654-8408-4422-8f28-37dbb4a77d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:36:08.443427 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.443367 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12e1654-8408-4422-8f28-37dbb4a77d49-maas-api-tls\") pod \"maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm\" (UID: \"a12e1654-8408-4422-8f28-37dbb4a77d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:36:08.443556 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.443445 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12e1654-8408-4422-8f28-37dbb4a77d49-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm\" (UID: \"a12e1654-8408-4422-8f28-37dbb4a77d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:36:08.552112 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.551275 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12e1654-8408-4422-8f28-37dbb4a77d49-metrics-certs\") pod \"maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm\" (UID: \"a12e1654-8408-4422-8f28-37dbb4a77d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:36:08.552112 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.551375 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86snq\" (UniqueName: \"kubernetes.io/projected/a12e1654-8408-4422-8f28-37dbb4a77d49-kube-api-access-86snq\") pod \"maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm\" (UID: \"a12e1654-8408-4422-8f28-37dbb4a77d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:36:08.552112 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.551455 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12e1654-8408-4422-8f28-37dbb4a77d49-maas-api-tls\") pod \"maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm\" (UID: \"a12e1654-8408-4422-8f28-37dbb4a77d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:36:08.552112 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.551519 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12e1654-8408-4422-8f28-37dbb4a77d49-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm\" (UID: \"a12e1654-8408-4422-8f28-37dbb4a77d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:36:08.562815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.552842 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12e1654-8408-4422-8f28-37dbb4a77d49-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm\" (UID: \"a12e1654-8408-4422-8f28-37dbb4a77d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:36:08.578062 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.574918 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12e1654-8408-4422-8f28-37dbb4a77d49-metrics-certs\") pod \"maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm\" (UID: \"a12e1654-8408-4422-8f28-37dbb4a77d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:36:08.616033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.611205 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12e1654-8408-4422-8f28-37dbb4a77d49-maas-api-tls\") pod \"maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm\" (UID: \"a12e1654-8408-4422-8f28-37dbb4a77d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:36:08.616033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.611976 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86snq\" (UniqueName: \"kubernetes.io/projected/a12e1654-8408-4422-8f28-37dbb4a77d49-kube-api-access-86snq\") pod \"maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm\" (UID: \"a12e1654-8408-4422-8f28-37dbb4a77d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:36:08.654076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:08.644695 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:36:09.011353 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:09.011318 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm"] Oct 30 19:36:09.022185 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:36:09.022149 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12e1654_8408_4422_8f28_37dbb4a77d49.slice/crio-088bea272ce13b5037ea9687769aef9432938fefac5536f4ce9fede1769ca7e7 WatchSource:0}: Error finding container 088bea272ce13b5037ea9687769aef9432938fefac5536f4ce9fede1769ca7e7: Status 404 returned error can't find the container with id 088bea272ce13b5037ea9687769aef9432938fefac5536f4ce9fede1769ca7e7 Oct 30 19:36:09.196497 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:09.196465 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" Oct 30 19:36:09.609051 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:36:09.608934 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:36:09.870117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:09.868510 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" event={"ID":"a12e1654-8408-4422-8f28-37dbb4a77d49","Type":"ContainerStarted","Data":"41b53d90605bc7ce76ddf51737987692818b1733dca0fab8651fb86fc8c63161"} Oct 30 19:36:09.870117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:09.868617 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" event={"ID":"a12e1654-8408-4422-8f28-37dbb4a77d49","Type":"ContainerStarted","Data":"088bea272ce13b5037ea9687769aef9432938fefac5536f4ce9fede1769ca7e7"} Oct 30 19:36:09.870117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:09.868674 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:36:09.909370 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:09.909311 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" podStartSLOduration=1.588833009 podStartE2EDuration="1.909296265s" totalImagesPullingTime="320.463256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:09.907116202 +0000 UTC m=+1922.981571561" watchObservedRunningTime="2026-10-30 19:36:09.909296265 +0000 UTC m=+1922.983751623" Oct 30 19:36:16.887546 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:16.887514 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:36:18.796112 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:18.796022 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" Oct 30 19:36:18.796530 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:18.796118 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" Oct 30 19:36:20.379376 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.379328 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw"] Oct 30 19:36:20.390159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.390127 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.415238 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.415197 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw"] Oct 30 19:36:20.421338 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.420182 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-19f646-openshift-default-dockercfg-ppvjk\"" Oct 30 19:36:20.436130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.436081 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcp6g\" (UniqueName: \"kubernetes.io/projected/78be6f9f-292a-4259-bb67-0d1dd1365325-kube-api-access-gcp6g\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.436321 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.436147 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78be6f9f-292a-4259-bb67-0d1dd1365325-istiod-ca-cert\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.436321 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.436259 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-podinfo\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.436418 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.436340 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-token\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.436497 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.436470 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-credential-socket\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.436555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.436539 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-workload-socket\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.436595 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.436566 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-data\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.436735 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.436722 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-workload-certs\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.436795 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.436781 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-envoy\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.568028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.565388 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-workload-certs\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.568028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.565471 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-envoy\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.568028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.565653 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcp6g\" (UniqueName: \"kubernetes.io/projected/78be6f9f-292a-4259-bb67-0d1dd1365325-kube-api-access-gcp6g\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.568028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.565733 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78be6f9f-292a-4259-bb67-0d1dd1365325-istiod-ca-cert\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.568028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.565841 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-podinfo\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.568028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.565944 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-token\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.568028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.565980 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-credential-socket\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.568028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.566033 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-workload-socket\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.568028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.566062 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-data\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.590027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.577020 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78be6f9f-292a-4259-bb67-0d1dd1365325-istiod-ca-cert\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.608032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.591619 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-data\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.608032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.591887 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-workload-certs\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.608032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.594168 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-credential-socket\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.608032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.598179 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-workload-socket\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.608032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.605709 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-envoy\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.660102 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.652281 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcp6g\" (UniqueName: \"kubernetes.io/projected/78be6f9f-292a-4259-bb67-0d1dd1365325-kube-api-access-gcp6g\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.673032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.660979 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-podinfo\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.673451 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.669722 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-token\") pod \"e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:20.722212 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:20.721791 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:21.218783 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:36:21.218750 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78be6f9f_292a_4259_bb67_0d1dd1365325.slice/crio-a0b86914ecc4be376f8a0da1d1f8d0fcbb9457c68e37b2c9655c7cd0ae3425ee WatchSource:0}: Error finding container a0b86914ecc4be376f8a0da1d1f8d0fcbb9457c68e37b2c9655c7cd0ae3425ee: Status 404 returned error can't find the container with id a0b86914ecc4be376f8a0da1d1f8d0fcbb9457c68e37b2c9655c7cd0ae3425ee Oct 30 19:36:21.220282 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:21.220254 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw"] Oct 30 19:36:21.222553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:21.222523 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:21.222631 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:21.222594 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:21.222631 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:21.222623 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:22.050179 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:22.050142 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" event={"ID":"78be6f9f-292a-4259-bb67-0d1dd1365325","Type":"ContainerStarted","Data":"430f8a4b7e0569c133f6151c6449a8334ed5743667888b300e06850b869c9471"} Oct 30 19:36:22.050602 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:22.050196 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" event={"ID":"78be6f9f-292a-4259-bb67-0d1dd1365325","Type":"ContainerStarted","Data":"a0b86914ecc4be376f8a0da1d1f8d0fcbb9457c68e37b2c9655c7cd0ae3425ee"} Oct 30 19:36:22.103123 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:22.103056 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" podStartSLOduration=2.102994212 podStartE2EDuration="2.102994212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:22.100877161 +0000 UTC m=+1935.175332521" watchObservedRunningTime="2026-10-30 19:36:22.102994212 +0000 UTC m=+1935.177449572" Oct 30 19:36:22.723452 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:22.723406 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:22.729596 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:22.729567 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:23.057967 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:23.057864 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:23.058834 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:23.058792 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:36:25.932628 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:25.929275 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp"] Oct 30 19:36:25.934992 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:25.934959 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:25.978685 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:25.978643 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp"] Oct 30 19:36:26.068387 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.068350 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.068592 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.068515 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d06c47f-e741-4465-a0bb-63b6544002da-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.068648 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.068616 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.068689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.068643 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d06c47f-e741-4465-a0bb-63b6544002da-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.068802 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.068764 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.068802 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.068797 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.068908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.068884 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d06c47f-e741-4465-a0bb-63b6544002da-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.068952 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.068931 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.068986 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.068952 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5db5l\" (UniqueName: \"kubernetes.io/projected/1d06c47f-e741-4465-a0bb-63b6544002da-kube-api-access-5db5l\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.170975 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.170927 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d06c47f-e741-4465-a0bb-63b6544002da-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.171206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.171081 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.171206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.171127 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d06c47f-e741-4465-a0bb-63b6544002da-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.171305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.171254 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.171305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.171289 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.171408 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.171361 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d06c47f-e741-4465-a0bb-63b6544002da-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.171465 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.171406 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.171465 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.171432 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5db5l\" (UniqueName: \"kubernetes.io/projected/1d06c47f-e741-4465-a0bb-63b6544002da-kube-api-access-5db5l\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.171580 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.171554 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.171920 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.171891 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.172146 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.172111 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.172146 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.171935 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.172325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.172192 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.172325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.172192 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d06c47f-e741-4465-a0bb-63b6544002da-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.174268 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.174241 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.174434 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.174407 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d06c47f-e741-4465-a0bb-63b6544002da-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.190756 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.190683 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5db5l\" (UniqueName: \"kubernetes.io/projected/1d06c47f-e741-4465-a0bb-63b6544002da-kube-api-access-5db5l\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.190887 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.190849 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d06c47f-e741-4465-a0bb-63b6544002da-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.296896 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.296856 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:26.709738 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.709689 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv"] Oct 30 19:36:26.716654 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.716623 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" Oct 30 19:36:26.767037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.764144 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-19f646\"" Oct 30 19:36:26.782421 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.782335 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-19f646-dockercfg-k7msk\"" Oct 30 19:36:26.905085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:26.828346 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fq9d\" (UniqueName: \"kubernetes.io/projected/0dbb629f-7a48-4553-abc8-46ec8c370a35-kube-api-access-5fq9d\") pod \"payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv\" (UID: \"0dbb629f-7a48-4553-abc8-46ec8c370a35\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" Oct 30 19:36:27.011253 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:27.011211 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbb629f-7a48-4553-abc8-46ec8c370a35-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv\" (UID: \"0dbb629f-7a48-4553-abc8-46ec8c370a35\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" Oct 30 19:36:27.017690 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:27.017586 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv"] Oct 30 19:36:27.117544 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:27.117499 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fq9d\" (UniqueName: \"kubernetes.io/projected/0dbb629f-7a48-4553-abc8-46ec8c370a35-kube-api-access-5fq9d\") pod \"payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv\" (UID: \"0dbb629f-7a48-4553-abc8-46ec8c370a35\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" Oct 30 19:36:27.118239 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:27.118200 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbb629f-7a48-4553-abc8-46ec8c370a35-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv\" (UID: \"0dbb629f-7a48-4553-abc8-46ec8c370a35\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" Oct 30 19:36:27.122599 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:27.122568 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbb629f-7a48-4553-abc8-46ec8c370a35-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv\" (UID: \"0dbb629f-7a48-4553-abc8-46ec8c370a35\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" Oct 30 19:36:27.162304 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:27.161891 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:27.162304 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:27.161994 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:27.162304 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:27.162051 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:27.171711 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:27.167532 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fq9d\" (UniqueName: \"kubernetes.io/projected/0dbb629f-7a48-4553-abc8-46ec8c370a35-kube-api-access-5fq9d\") pod \"payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv\" (UID: \"0dbb629f-7a48-4553-abc8-46ec8c370a35\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" Oct 30 19:36:27.200495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:27.200439 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" event={"ID":"1d06c47f-e741-4465-a0bb-63b6544002da","Type":"ContainerStarted","Data":"fc701028a455e4a6705f4a9ce57c25a7447be8b49502ef5ff5d4a6ac0a05832e"} Oct 30 19:36:27.200744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:27.200516 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp"] Oct 30 19:36:27.458085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:27.454307 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" Oct 30 19:36:27.821834 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:36:27.820865 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbb629f_7a48_4553_abc8_46ec8c370a35.slice/crio-7ca702c8ab4734fdd4d16543ad51954bce7d7271c1c5d178fb447ae5f8e6f607 WatchSource:0}: Error finding container 7ca702c8ab4734fdd4d16543ad51954bce7d7271c1c5d178fb447ae5f8e6f607: Status 404 returned error can't find the container with id 7ca702c8ab4734fdd4d16543ad51954bce7d7271c1c5d178fb447ae5f8e6f607 Oct 30 19:36:27.824088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:27.823892 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv"] Oct 30 19:36:28.208193 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:28.208157 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" event={"ID":"1d06c47f-e741-4465-a0bb-63b6544002da","Type":"ContainerStarted","Data":"578dee23a51f24b7f45024b7c7b561f8f253d4fd9163925b4cb0943a40aab0e9"} Oct 30 19:36:28.210091 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:28.210064 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" event={"ID":"0dbb629f-7a48-4553-abc8-46ec8c370a35","Type":"ContainerStarted","Data":"e23538c76db4ba34576b3ba5d19cf894c0ad11abce0464af482c6aaedf40414f"} Oct 30 19:36:28.210203 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:28.210107 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" event={"ID":"0dbb629f-7a48-4553-abc8-46ec8c370a35","Type":"ContainerStarted","Data":"7ca702c8ab4734fdd4d16543ad51954bce7d7271c1c5d178fb447ae5f8e6f607"} Oct 30 19:36:28.210203 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:28.210139 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" Oct 30 19:36:28.297323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:28.297290 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:28.299719 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:28.299681 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:36:28.299899 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:28.299747 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" podUID="1d06c47f-e741-4465-a0bb-63b6544002da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 19:36:28.303568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:28.303515 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" podStartSLOduration=3.303500534 podStartE2EDuration="3.303500534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:28.289490861 +0000 UTC m=+1941.363946221" watchObservedRunningTime="2026-10-30 19:36:28.303500534 +0000 UTC m=+1941.377955893" Oct 30 19:36:28.329019 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:28.328957 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" podStartSLOduration=2.328942601 podStartE2EDuration="2.328942601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:28.325933019 +0000 UTC m=+1941.400388379" watchObservedRunningTime="2026-10-30 19:36:28.328942601 +0000 UTC m=+1941.403397980" Oct 30 19:36:29.298635 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:29.298600 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:36:29.299078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:29.298670 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" podUID="1d06c47f-e741-4465-a0bb-63b6544002da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 19:36:30.306083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:30.300473 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:36:30.306083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:30.300539 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" podUID="1d06c47f-e741-4465-a0bb-63b6544002da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 19:36:31.298496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:31.298454 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:36:31.298664 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:31.298516 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" podUID="1d06c47f-e741-4465-a0bb-63b6544002da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 19:36:32.305035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:32.304631 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:36:32.305035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:32.304695 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" podUID="1d06c47f-e741-4465-a0bb-63b6544002da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 19:36:33.297916 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:33.297877 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:36:33.298124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:33.297945 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" podUID="1d06c47f-e741-4465-a0bb-63b6544002da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 19:36:33.897176 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:33.897139 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9"] Oct 30 19:36:33.902316 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:33.902295 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:36:33.929608 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:33.929573 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d4cb13-serving-cert\"" Oct 30 19:36:33.933634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:33.933602 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9"] Oct 30 19:36:34.008279 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:34.008232 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eca5e1e9-76eb-4731-8041-1f521637e214-metrics-certs\") pod \"maas-api-e2e-api-a-d4cb13-65fb7749-htkc9\" (UID: \"eca5e1e9-76eb-4731-8041-1f521637e214\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:36:34.008569 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:34.008544 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-975jz\" (UniqueName: \"kubernetes.io/projected/eca5e1e9-76eb-4731-8041-1f521637e214-kube-api-access-975jz\") pod \"maas-api-e2e-api-a-d4cb13-65fb7749-htkc9\" (UID: \"eca5e1e9-76eb-4731-8041-1f521637e214\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:36:34.008657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:34.008619 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eca5e1e9-76eb-4731-8041-1f521637e214-openshift-service-ca\") pod \"maas-api-e2e-api-a-d4cb13-65fb7749-htkc9\" (UID: \"eca5e1e9-76eb-4731-8041-1f521637e214\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:36:34.008848 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:34.008816 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eca5e1e9-76eb-4731-8041-1f521637e214-maas-api-tls\") pod \"maas-api-e2e-api-a-d4cb13-65fb7749-htkc9\" (UID: \"eca5e1e9-76eb-4731-8041-1f521637e214\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:36:34.182334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:34.169544 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-975jz\" (UniqueName: \"kubernetes.io/projected/eca5e1e9-76eb-4731-8041-1f521637e214-kube-api-access-975jz\") pod \"maas-api-e2e-api-a-d4cb13-65fb7749-htkc9\" (UID: \"eca5e1e9-76eb-4731-8041-1f521637e214\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:36:34.182334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:34.169643 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eca5e1e9-76eb-4731-8041-1f521637e214-openshift-service-ca\") pod \"maas-api-e2e-api-a-d4cb13-65fb7749-htkc9\" (UID: \"eca5e1e9-76eb-4731-8041-1f521637e214\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:36:34.182334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:34.169822 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eca5e1e9-76eb-4731-8041-1f521637e214-maas-api-tls\") pod \"maas-api-e2e-api-a-d4cb13-65fb7749-htkc9\" (UID: \"eca5e1e9-76eb-4731-8041-1f521637e214\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:36:34.182334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:34.172523 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eca5e1e9-76eb-4731-8041-1f521637e214-openshift-service-ca\") pod \"maas-api-e2e-api-a-d4cb13-65fb7749-htkc9\" (UID: \"eca5e1e9-76eb-4731-8041-1f521637e214\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:36:34.182334 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:36:34.172942 5047 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-serving-cert: secret "maas-api-e2e-api-a-d4cb13-serving-cert" not found Oct 30 19:36:34.182334 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:36:34.173024 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca5e1e9-76eb-4731-8041-1f521637e214-maas-api-tls podName:eca5e1e9-76eb-4731-8041-1f521637e214 nodeName:}" failed. No retries permitted until 2026-10-30 19:36:34.672987651 +0000 UTC m=+1947.747443006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eca5e1e9-76eb-4731-8041-1f521637e214-maas-api-tls") pod "maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" (UID: "eca5e1e9-76eb-4731-8041-1f521637e214") : secret "maas-api-e2e-api-a-d4cb13-serving-cert" not found Oct 30 19:36:34.183197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:34.182840 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eca5e1e9-76eb-4731-8041-1f521637e214-metrics-certs\") pod \"maas-api-e2e-api-a-d4cb13-65fb7749-htkc9\" (UID: \"eca5e1e9-76eb-4731-8041-1f521637e214\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:36:34.217822 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:34.217652 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eca5e1e9-76eb-4731-8041-1f521637e214-metrics-certs\") pod \"maas-api-e2e-api-a-d4cb13-65fb7749-htkc9\" (UID: \"eca5e1e9-76eb-4731-8041-1f521637e214\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:36:34.251053 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:34.242623 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-975jz\" (UniqueName: \"kubernetes.io/projected/eca5e1e9-76eb-4731-8041-1f521637e214-kube-api-access-975jz\") pod \"maas-api-e2e-api-a-d4cb13-65fb7749-htkc9\" (UID: \"eca5e1e9-76eb-4731-8041-1f521637e214\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:36:34.378387 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:34.378347 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:34.378976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:34.378950 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:34.380282 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:34.380258 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:36:34.693998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:34.693959 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eca5e1e9-76eb-4731-8041-1f521637e214-maas-api-tls\") pod \"maas-api-e2e-api-a-d4cb13-65fb7749-htkc9\" (UID: \"eca5e1e9-76eb-4731-8041-1f521637e214\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:36:34.696697 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:34.696667 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eca5e1e9-76eb-4731-8041-1f521637e214-maas-api-tls\") pod \"maas-api-e2e-api-a-d4cb13-65fb7749-htkc9\" (UID: \"eca5e1e9-76eb-4731-8041-1f521637e214\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:36:34.834130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:34.832366 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:36:35.186022 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:36:35.185968 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca5e1e9_76eb_4731_8041_1f521637e214.slice/crio-3c4c5c183da5c86d77474a183e45867fc0a0c86558ca9a4d11a7afc5493afcc0 WatchSource:0}: Error finding container 3c4c5c183da5c86d77474a183e45867fc0a0c86558ca9a4d11a7afc5493afcc0: Status 404 returned error can't find the container with id 3c4c5c183da5c86d77474a183e45867fc0a0c86558ca9a4d11a7afc5493afcc0 Oct 30 19:36:35.186436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:35.186155 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9"] Oct 30 19:36:35.439616 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:35.439576 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" event={"ID":"eca5e1e9-76eb-4731-8041-1f521637e214","Type":"ContainerStarted","Data":"3c4c5c183da5c86d77474a183e45867fc0a0c86558ca9a4d11a7afc5493afcc0"} Oct 30 19:36:36.449814 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:36.449772 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" event={"ID":"eca5e1e9-76eb-4731-8041-1f521637e214","Type":"ContainerStarted","Data":"ea3b004fed94f7566f962ec9ac5690df8996ddcad7988a442145c5b22f48dccc"} Oct 30 19:36:36.450264 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:36.449869 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:36:36.499504 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:36.499433 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" podStartSLOduration=3.130146313 podStartE2EDuration="3.499410026s" totalImagesPullingTime="369.263713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:36.494564466 +0000 UTC m=+1949.569019825" watchObservedRunningTime="2026-10-30 19:36:36.499410026 +0000 UTC m=+1949.573865387" Oct 30 19:36:37.161193 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:37.161151 5047 patch_prober.go:29] interesting pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:36:37.161382 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:37.161245 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" podUID="22ab4a50-e3a3-4312-8de3-4c9656b724de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:39.222465 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:39.222439 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" Oct 30 19:36:39.481985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:39.481876 5047 generic.go:356] "Generic (PLEG): container finished" podID="22ab4a50-e3a3-4312-8de3-4c9656b724de" containerID="d3a04500d82cb2b769d0cfe5acc230928b4320dca934df79bf853ee2b17aaaae" exitCode=0 Oct 30 19:36:39.481985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:39.481944 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" event={"ID":"22ab4a50-e3a3-4312-8de3-4c9656b724de","Type":"ContainerDied","Data":"d3a04500d82cb2b769d0cfe5acc230928b4320dca934df79bf853ee2b17aaaae"} Oct 30 19:36:39.482600 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:39.482583 5047 scope.go:122] "RemoveContainer" containerID="d3a04500d82cb2b769d0cfe5acc230928b4320dca934df79bf853ee2b17aaaae" Oct 30 19:36:39.484581 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:39.484551 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:39.484685 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:39.484624 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:39.484685 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:39.484655 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:39.615456 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:36:39.615413 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:36:40.493423 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:40.493384 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" event={"ID":"22ab4a50-e3a3-4312-8de3-4c9656b724de","Type":"ContainerStarted","Data":"4b1b09424cf99cc1f976febdd1ab41ef1b30b1b430b77e8af4b4c568aff4fb01"} Oct 30 19:36:41.453031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:41.452623 5047 patch_prober.go:29] interesting pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:36:41.453031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:41.452699 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" podUID="22ab4a50-e3a3-4312-8de3-4c9656b724de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 19:36:41.453315 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:41.453070 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:36:42.459485 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:42.459445 5047 patch_prober.go:29] interesting pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:36:42.459998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:42.459521 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" podUID="22ab4a50-e3a3-4312-8de3-4c9656b724de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 19:36:42.466939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:42.466912 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:36:43.452515 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:43.452474 5047 patch_prober.go:29] interesting pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:36:43.452823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:43.452791 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" podUID="22ab4a50-e3a3-4312-8de3-4c9656b724de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 19:36:44.453598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:44.453555 5047 patch_prober.go:29] interesting pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:36:44.454257 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:44.454220 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" podUID="22ab4a50-e3a3-4312-8de3-4c9656b724de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 19:36:44.667959 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:44.667919 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw"] Oct 30 19:36:44.668275 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:44.668243 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" podUID="b64318a8-a330-4651-8b72-726f3b31fce3" containerName="maas-api" containerID="cri-o://6c869e88f2b1517c71add4501fc9bd3936454c3279058f7b1aae1ff46b7a785a" gracePeriod=30 Oct 30 19:36:45.423466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.423434 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:36:45.457319 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.454477 5047 patch_prober.go:29] interesting pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:36:45.457319 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.454539 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" podUID="22ab4a50-e3a3-4312-8de3-4c9656b724de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 19:36:45.540376 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.536092 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b64318a8-a330-4651-8b72-726f3b31fce3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b64318a8-a330-4651-8b72-726f3b31fce3-metrics-certs\") pod \"b64318a8-a330-4651-8b72-726f3b31fce3\" (UID: \"b64318a8-a330-4651-8b72-726f3b31fce3\") " Oct 30 19:36:45.540376 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.536232 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64318a8-a330-4651-8b72-726f3b31fce3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b64318a8-a330-4651-8b72-726f3b31fce3-openshift-service-ca\") pod \"b64318a8-a330-4651-8b72-726f3b31fce3\" (UID: \"b64318a8-a330-4651-8b72-726f3b31fce3\") " Oct 30 19:36:45.540376 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.536373 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64318a8-a330-4651-8b72-726f3b31fce3-kube-api-access-ntpx6\" (UniqueName: \"kubernetes.io/projected/b64318a8-a330-4651-8b72-726f3b31fce3-kube-api-access-ntpx6\") pod \"b64318a8-a330-4651-8b72-726f3b31fce3\" (UID: \"b64318a8-a330-4651-8b72-726f3b31fce3\") " Oct 30 19:36:45.540376 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.536412 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b64318a8-a330-4651-8b72-726f3b31fce3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b64318a8-a330-4651-8b72-726f3b31fce3-maas-api-tls\") pod \"b64318a8-a330-4651-8b72-726f3b31fce3\" (UID: \"b64318a8-a330-4651-8b72-726f3b31fce3\") " Oct 30 19:36:45.588465 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.562200 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64318a8-a330-4651-8b72-726f3b31fce3-openshift-service-ca" pod "b64318a8-a330-4651-8b72-726f3b31fce3" (UID: "b64318a8-a330-4651-8b72-726f3b31fce3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:45.611824 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.611495 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b64318a8-a330-4651-8b72-726f3b31fce3-maas-api-tls" pod "b64318a8-a330-4651-8b72-726f3b31fce3" (UID: "b64318a8-a330-4651-8b72-726f3b31fce3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:45.629317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.629281 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b64318a8-a330-4651-8b72-726f3b31fce3-metrics-certs" pod "b64318a8-a330-4651-8b72-726f3b31fce3" (UID: "b64318a8-a330-4651-8b72-726f3b31fce3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:45.635154 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.630992 5047 generic.go:356] "Generic (PLEG): container finished" podID="b64318a8-a330-4651-8b72-726f3b31fce3" containerID="6c869e88f2b1517c71add4501fc9bd3936454c3279058f7b1aae1ff46b7a785a" exitCode=0 Oct 30 19:36:45.635525 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.631102 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" event={"ID":"b64318a8-a330-4651-8b72-726f3b31fce3","Type":"ContainerDied","Data":"6c869e88f2b1517c71add4501fc9bd3936454c3279058f7b1aae1ff46b7a785a"} Oct 30 19:36:45.641634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.631336 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" Oct 30 19:36:45.641969 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.641876 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw" event={"ID":"b64318a8-a330-4651-8b72-726f3b31fce3","Type":"ContainerDied","Data":"baca58a13057fe7e043630b26468b31a1811a01f9a778117ccd3f55c967eb7a8"} Oct 30 19:36:45.642164 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.641932 5047 scope.go:122] "RemoveContainer" containerID="6c869e88f2b1517c71add4501fc9bd3936454c3279058f7b1aae1ff46b7a785a" Oct 30 19:36:45.663234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.660451 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b64318a8-a330-4651-8b72-726f3b31fce3-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:36:45.663234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.660487 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b64318a8-a330-4651-8b72-726f3b31fce3-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:36:45.663234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.660502 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b64318a8-a330-4651-8b72-726f3b31fce3-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:36:45.672564 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.672536 5047 scope.go:122] "RemoveContainer" containerID="6c869e88f2b1517c71add4501fc9bd3936454c3279058f7b1aae1ff46b7a785a" Oct 30 19:36:45.673645 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:36:45.673560 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c869e88f2b1517c71add4501fc9bd3936454c3279058f7b1aae1ff46b7a785a\": container with ID starting with 6c869e88f2b1517c71add4501fc9bd3936454c3279058f7b1aae1ff46b7a785a not found: ID does not exist" containerID="6c869e88f2b1517c71add4501fc9bd3936454c3279058f7b1aae1ff46b7a785a" Oct 30 19:36:45.673645 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.673607 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c869e88f2b1517c71add4501fc9bd3936454c3279058f7b1aae1ff46b7a785a"} err="failed to get container status \"6c869e88f2b1517c71add4501fc9bd3936454c3279058f7b1aae1ff46b7a785a\": rpc error: code = NotFound desc = could not find container \"6c869e88f2b1517c71add4501fc9bd3936454c3279058f7b1aae1ff46b7a785a\": container with ID starting with 6c869e88f2b1517c71add4501fc9bd3936454c3279058f7b1aae1ff46b7a785a not found: ID does not exist" Oct 30 19:36:45.686171 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.685833 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64318a8-a330-4651-8b72-726f3b31fce3-kube-api-access-ntpx6" pod "b64318a8-a330-4651-8b72-726f3b31fce3" (UID: "b64318a8-a330-4651-8b72-726f3b31fce3"). InnerVolumeSpecName "kube-api-access-ntpx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:45.770029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:45.769519 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntpx6\" (UniqueName: \"kubernetes.io/projected/b64318a8-a330-4651-8b72-726f3b31fce3-kube-api-access-ntpx6\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:36:46.147078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:46.146986 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw"] Oct 30 19:36:46.184040 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:46.182904 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-891d6b-76b4fbd666-d4whw"] Oct 30 19:36:46.451617 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:46.451579 5047 patch_prober.go:29] interesting pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:36:46.451926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:46.451896 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" podUID="22ab4a50-e3a3-4312-8de3-4c9656b724de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 19:36:47.054079 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:47.054040 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64318a8-a330-4651-8b72-726f3b31fce3" path="/var/lib/kubelet/pods/b64318a8-a330-4651-8b72-726f3b31fce3/volumes" Oct 30 19:36:47.454915 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:47.454884 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:36:47.455381 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:47.455368 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:36:47.456045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:47.456000 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:36:56.136590 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.136548 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc"] Oct 30 19:36:56.137615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.137591 5047 cpu_manager.go:414] "containerMap: removing container" podUID="b64318a8-a330-4651-8b72-726f3b31fce3" containerName="maas-api" Oct 30 19:36:56.137615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.137612 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64318a8-a330-4651-8b72-726f3b31fce3" containerName="maas-api" Oct 30 19:36:56.137790 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.137755 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64318a8-a330-4651-8b72-726f3b31fce3" containerName="maas-api" Oct 30 19:36:56.141747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.141721 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.162723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.162683 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc"] Oct 30 19:36:56.166783 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.166749 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f4ab4a\"/\"test-model-f4ab4a-kserve-self-signed-certs\"" Oct 30 19:36:56.166783 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.166756 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f4ab4a\"/\"default-dockercfg-b2xzb\"" Oct 30 19:36:56.251895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.251865 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f4ab4a\"/\"kube-root-ca.crt\"" Oct 30 19:36:56.252103 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.251917 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f4ab4a\"/\"openshift-service-ca.crt\"" Oct 30 19:36:56.266798 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.266762 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vkts\" (UniqueName: \"kubernetes.io/projected/68ea348d-f2bf-4ee5-9062-9649b71be4c3-kube-api-access-4vkts\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.266979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.266906 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-home\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.266979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.266971 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-dshm\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.267146 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.267127 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68ea348d-f2bf-4ee5-9062-9649b71be4c3-tls-certs\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.267295 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.267277 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-model-cache\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.267344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.267321 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-tmp-dir\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.369282 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.369243 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-home\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.369480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.369308 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-dshm\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.369480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.369421 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68ea348d-f2bf-4ee5-9062-9649b71be4c3-tls-certs\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.369615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.369573 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-model-cache\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.369674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.369651 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-tmp-dir\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.369736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.369718 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-home\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.369832 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.369792 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vkts\" (UniqueName: \"kubernetes.io/projected/68ea348d-f2bf-4ee5-9062-9649b71be4c3-kube-api-access-4vkts\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.369933 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.369900 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-model-cache\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.370106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.369965 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-tmp-dir\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.371662 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.371642 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-dshm\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.371987 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.371963 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68ea348d-f2bf-4ee5-9062-9649b71be4c3-tls-certs\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.392318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.392255 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vkts\" (UniqueName: \"kubernetes.io/projected/68ea348d-f2bf-4ee5-9062-9649b71be4c3-kube-api-access-4vkts\") pod \"test-model-f4ab4a-kserve-64fcbc75f9-fdkgc\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.476710 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.472878 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:36:56.917857 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:36:56.917780 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68ea348d_f2bf_4ee5_9062_9649b71be4c3.slice/crio-6b81092ee7c7eaaeb7cb1f555ed9d077efc9e373a0c45ff6a72421ebf75fd647 WatchSource:0}: Error finding container 6b81092ee7c7eaaeb7cb1f555ed9d077efc9e373a0c45ff6a72421ebf75fd647: Status 404 returned error can't find the container with id 6b81092ee7c7eaaeb7cb1f555ed9d077efc9e373a0c45ff6a72421ebf75fd647 Oct 30 19:36:56.918796 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:56.918770 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc"] Oct 30 19:36:57.745705 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:57.745669 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" event={"ID":"68ea348d-f2bf-4ee5-9062-9649b71be4c3","Type":"ContainerStarted","Data":"9577a0d4bd3235a7403ae676b856845ab28263d5c81a0f87a549f80e12a238ac"} Oct 30 19:36:57.746143 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:57.745713 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" event={"ID":"68ea348d-f2bf-4ee5-9062-9649b71be4c3","Type":"ContainerStarted","Data":"6b81092ee7c7eaaeb7cb1f555ed9d077efc9e373a0c45ff6a72421ebf75fd647"} Oct 30 19:36:57.783936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:57.783885 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" podStartSLOduration=1.7838717320000002 podStartE2EDuration="1.783871732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:57.783608762 +0000 UTC m=+1970.858064121" watchObservedRunningTime="2026-10-30 19:36:57.783871732 +0000 UTC m=+1970.858327092" Oct 30 19:36:58.473781 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:36:58.473734 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:37:03.746805 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:03.746766 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp"] Oct 30 19:37:03.751966 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:03.751944 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:37:03.776045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:03.774563 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-19f646-serving-cert\"" Oct 30 19:37:03.788689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:03.787195 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp"] Oct 30 19:37:03.796530 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:03.796494 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp\" (UID: \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:37:03.796707 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:03.796635 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-metrics-certs\") pod \"maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp\" (UID: \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:37:03.796787 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:03.796764 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h98jc\" (UniqueName: \"kubernetes.io/projected/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-kube-api-access-h98jc\") pod \"maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp\" (UID: \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:37:03.796919 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:03.796904 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-maas-api-tls\") pod \"maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp\" (UID: \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:37:03.899041 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:03.898979 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp\" (UID: \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:37:03.899231 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:03.899073 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-metrics-certs\") pod \"maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp\" (UID: \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:37:03.899231 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:03.899155 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h98jc\" (UniqueName: \"kubernetes.io/projected/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-kube-api-access-h98jc\") pod \"maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp\" (UID: \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:37:03.899328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:03.899274 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-maas-api-tls\") pod \"maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp\" (UID: \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:37:03.899723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:03.899698 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp\" (UID: \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:37:03.901620 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:03.901596 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-metrics-certs\") pod \"maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp\" (UID: \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:37:03.901755 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:03.901734 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-maas-api-tls\") pod \"maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp\" (UID: \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:37:03.923109 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:03.923079 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h98jc\" (UniqueName: \"kubernetes.io/projected/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-kube-api-access-h98jc\") pod \"maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp\" (UID: \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:37:04.077088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:04.073368 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:37:04.474749 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:37:04.474719 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0c1a23_136f_4fc5_9c09_ab162557fb8f.slice/crio-72b2f1681f374b7e5c88b3dd8476f85e2e1173c80d8bca460fc3ff22f12f8bc9 WatchSource:0}: Error finding container 72b2f1681f374b7e5c88b3dd8476f85e2e1173c80d8bca460fc3ff22f12f8bc9: Status 404 returned error can't find the container with id 72b2f1681f374b7e5c88b3dd8476f85e2e1173c80d8bca460fc3ff22f12f8bc9 Oct 30 19:37:04.476166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:04.476143 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp"] Oct 30 19:37:04.834720 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:04.834676 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" event={"ID":"0b0c1a23-136f-4fc5-9c09-ab162557fb8f","Type":"ContainerStarted","Data":"72b2f1681f374b7e5c88b3dd8476f85e2e1173c80d8bca460fc3ff22f12f8bc9"} Oct 30 19:37:05.845398 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:05.845352 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" event={"ID":"0b0c1a23-136f-4fc5-9c09-ab162557fb8f","Type":"ContainerStarted","Data":"a2157133950e09abbaa84af249277d5bb9bfd074e0573252369bf0c7f933bf5e"} Oct 30 19:37:05.845806 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:05.845470 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:37:05.887956 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:05.887890 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" podStartSLOduration=2.390946722 podStartE2EDuration="2.887875029s" totalImagesPullingTime="496.928307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:05.883708053 +0000 UTC m=+1978.958163413" watchObservedRunningTime="2026-10-30 19:37:05.887875029 +0000 UTC m=+1978.962330387" Oct 30 19:37:06.474036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:06.473991 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:37:06.488640 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:06.488607 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:37:06.914710 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:06.914435 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:37:09.604557 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:37:09.604521 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:37:11.892664 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:11.892632 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:37:13.091136 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:13.091072 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns"] Oct 30 19:37:13.091514 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:13.091338 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" podUID="faec438d-9b43-450a-b39e-e6eabb6e0b98" containerName="istio-proxy" containerID="cri-o://62a8abd4e9dbd99351780fc8efd3fcb0d6cb4bdadd99c173eac5fde05bad7d73" gracePeriod=30 Oct 30 19:37:18.487466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.487407 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:37:18.609148 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.609110 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faec438d-9b43-450a-b39e-e6eabb6e0b98-kube-api-access-df4b5\" (UniqueName: \"kubernetes.io/projected/faec438d-9b43-450a-b39e-e6eabb6e0b98-kube-api-access-df4b5\") pod \"faec438d-9b43-450a-b39e-e6eabb6e0b98\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " Oct 30 19:37:18.609336 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.609188 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-credential-socket\") pod \"faec438d-9b43-450a-b39e-e6eabb6e0b98\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " Oct 30 19:37:18.609408 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.609386 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-data\") pod \"faec438d-9b43-450a-b39e-e6eabb6e0b98\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " Oct 30 19:37:18.609466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.609441 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-workload-socket\") pod \"faec438d-9b43-450a-b39e-e6eabb6e0b98\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " Oct 30 19:37:18.609526 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.609487 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-credential-socket" pod "faec438d-9b43-450a-b39e-e6eabb6e0b98" (UID: "faec438d-9b43-450a-b39e-e6eabb6e0b98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.609583 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.609528 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-token\" (UniqueName: \"kubernetes.io/projected/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-token\") pod \"faec438d-9b43-450a-b39e-e6eabb6e0b98\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " Oct 30 19:37:18.609583 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.609567 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-podinfo\") pod \"faec438d-9b43-450a-b39e-e6eabb6e0b98\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " Oct 30 19:37:18.609745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.609628 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faec438d-9b43-450a-b39e-e6eabb6e0b98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faec438d-9b43-450a-b39e-e6eabb6e0b98-istiod-ca-cert\") pod \"faec438d-9b43-450a-b39e-e6eabb6e0b98\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " Oct 30 19:37:18.609745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.609677 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-data" pod "faec438d-9b43-450a-b39e-e6eabb6e0b98" (UID: "faec438d-9b43-450a-b39e-e6eabb6e0b98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.609745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.609686 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-workload-certs\") pod \"faec438d-9b43-450a-b39e-e6eabb6e0b98\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " Oct 30 19:37:18.609906 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.609757 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-workload-socket" pod "faec438d-9b43-450a-b39e-e6eabb6e0b98" (UID: "faec438d-9b43-450a-b39e-e6eabb6e0b98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.609906 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.609870 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-envoy\") pod \"faec438d-9b43-450a-b39e-e6eabb6e0b98\" (UID: \"faec438d-9b43-450a-b39e-e6eabb6e0b98\") " Oct 30 19:37:18.610044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.609941 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-workload-certs" pod "faec438d-9b43-450a-b39e-e6eabb6e0b98" (UID: "faec438d-9b43-450a-b39e-e6eabb6e0b98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.610142 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.610114 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faec438d-9b43-450a-b39e-e6eabb6e0b98-istiod-ca-cert" pod "faec438d-9b43-450a-b39e-e6eabb6e0b98" (UID: "faec438d-9b43-450a-b39e-e6eabb6e0b98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:18.610597 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.610575 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:37:18.610701 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.610616 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:37:18.610701 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.610625 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:37:18.610701 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.610634 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faec438d-9b43-450a-b39e-e6eabb6e0b98-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:37:18.610701 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.610642 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:37:18.611916 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.611884 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-envoy" pod "faec438d-9b43-450a-b39e-e6eabb6e0b98" (UID: "faec438d-9b43-450a-b39e-e6eabb6e0b98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.611995 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.611972 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-podinfo" pod "faec438d-9b43-450a-b39e-e6eabb6e0b98" (UID: "faec438d-9b43-450a-b39e-e6eabb6e0b98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:18.612305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.612281 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-token" pod "faec438d-9b43-450a-b39e-e6eabb6e0b98" (UID: "faec438d-9b43-450a-b39e-e6eabb6e0b98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:18.612356 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.612318 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faec438d-9b43-450a-b39e-e6eabb6e0b98-kube-api-access-df4b5" pod "faec438d-9b43-450a-b39e-e6eabb6e0b98" (UID: "faec438d-9b43-450a-b39e-e6eabb6e0b98"). InnerVolumeSpecName "kube-api-access-df4b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:18.712569 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.712478 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:37:18.712569 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.712518 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:37:18.712569 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.712534 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faec438d-9b43-450a-b39e-e6eabb6e0b98-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:37:18.712569 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:18.712547 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df4b5\" (UniqueName: \"kubernetes.io/projected/faec438d-9b43-450a-b39e-e6eabb6e0b98-kube-api-access-df4b5\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:37:19.104406 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.104377 5047 generic.go:356] "Generic (PLEG): container finished" podID="90f0e672-c84b-4721-9a09-363ffdf05947" containerID="b1250f61e90a56e35aa35cdae53794312ff17d973da85d4631f4b2a7abad7105" exitCode=0 Oct 30 19:37:19.104585 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.104451 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" event={"ID":"90f0e672-c84b-4721-9a09-363ffdf05947","Type":"ContainerDied","Data":"b1250f61e90a56e35aa35cdae53794312ff17d973da85d4631f4b2a7abad7105"} Oct 30 19:37:19.104955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.104934 5047 scope.go:122] "RemoveContainer" containerID="b1250f61e90a56e35aa35cdae53794312ff17d973da85d4631f4b2a7abad7105" Oct 30 19:37:19.106419 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.106399 5047 generic.go:356] "Generic (PLEG): container finished" podID="faec438d-9b43-450a-b39e-e6eabb6e0b98" containerID="62a8abd4e9dbd99351780fc8efd3fcb0d6cb4bdadd99c173eac5fde05bad7d73" exitCode=0 Oct 30 19:37:19.106518 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.106432 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" event={"ID":"faec438d-9b43-450a-b39e-e6eabb6e0b98","Type":"ContainerDied","Data":"62a8abd4e9dbd99351780fc8efd3fcb0d6cb4bdadd99c173eac5fde05bad7d73"} Oct 30 19:37:19.106518 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.106458 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" Oct 30 19:37:19.106518 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.106466 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns" event={"ID":"faec438d-9b43-450a-b39e-e6eabb6e0b98","Type":"ContainerDied","Data":"117a75b4281c5925f7a2c876800a2b0bef635eb83b196f03989d635627261caa"} Oct 30 19:37:19.106518 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.106488 5047 scope.go:122] "RemoveContainer" containerID="62a8abd4e9dbd99351780fc8efd3fcb0d6cb4bdadd99c173eac5fde05bad7d73" Oct 30 19:37:19.107215 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.107181 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:19.107320 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.107260 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:19.107375 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.107323 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:19.118239 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.118215 5047 scope.go:122] "RemoveContainer" containerID="d649a5636675450846e4a3effd0c5f3d2c9efd4b5b1d444076e3d99ee5b21059" Oct 30 19:37:19.129330 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.129313 5047 scope.go:122] "RemoveContainer" containerID="62a8abd4e9dbd99351780fc8efd3fcb0d6cb4bdadd99c173eac5fde05bad7d73" Oct 30 19:37:19.129604 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:37:19.129579 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a8abd4e9dbd99351780fc8efd3fcb0d6cb4bdadd99c173eac5fde05bad7d73\": container with ID starting with 62a8abd4e9dbd99351780fc8efd3fcb0d6cb4bdadd99c173eac5fde05bad7d73 not found: ID does not exist" containerID="62a8abd4e9dbd99351780fc8efd3fcb0d6cb4bdadd99c173eac5fde05bad7d73" Oct 30 19:37:19.129672 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.129610 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a8abd4e9dbd99351780fc8efd3fcb0d6cb4bdadd99c173eac5fde05bad7d73"} err="failed to get container status \"62a8abd4e9dbd99351780fc8efd3fcb0d6cb4bdadd99c173eac5fde05bad7d73\": rpc error: code = NotFound desc = could not find container \"62a8abd4e9dbd99351780fc8efd3fcb0d6cb4bdadd99c173eac5fde05bad7d73\": container with ID starting with 62a8abd4e9dbd99351780fc8efd3fcb0d6cb4bdadd99c173eac5fde05bad7d73 not found: ID does not exist" Oct 30 19:37:19.129672 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.129626 5047 scope.go:122] "RemoveContainer" containerID="d649a5636675450846e4a3effd0c5f3d2c9efd4b5b1d444076e3d99ee5b21059" Oct 30 19:37:19.129853 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:37:19.129837 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d649a5636675450846e4a3effd0c5f3d2c9efd4b5b1d444076e3d99ee5b21059\": container with ID starting with d649a5636675450846e4a3effd0c5f3d2c9efd4b5b1d444076e3d99ee5b21059 not found: ID does not exist" containerID="d649a5636675450846e4a3effd0c5f3d2c9efd4b5b1d444076e3d99ee5b21059" Oct 30 19:37:19.129950 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.129857 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d649a5636675450846e4a3effd0c5f3d2c9efd4b5b1d444076e3d99ee5b21059"} err="failed to get container status \"d649a5636675450846e4a3effd0c5f3d2c9efd4b5b1d444076e3d99ee5b21059\": rpc error: code = NotFound desc = could not find container \"d649a5636675450846e4a3effd0c5f3d2c9efd4b5b1d444076e3d99ee5b21059\": container with ID starting with d649a5636675450846e4a3effd0c5f3d2c9efd4b5b1d444076e3d99ee5b21059 not found: ID does not exist" Oct 30 19:37:19.205638 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.203278 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns"] Oct 30 19:37:19.234804 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:19.234562 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-891d6b-openshift-default-798fd7ccdf-zgjns"] Oct 30 19:37:20.176055 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:20.175974 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" event={"ID":"90f0e672-c84b-4721-9a09-363ffdf05947","Type":"ContainerStarted","Data":"a44b94fa2a2c7c29852894542ad33890811a80f30951ab50a8e21b0185f7618e"} Oct 30 19:37:20.255708 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:20.255659 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:37:20.256310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:20.256282 5047 patch_prober.go:29] interesting pod/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 19:37:20.256634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:20.256600 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" podUID="90f0e672-c84b-4721-9a09-363ffdf05947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 19:37:21.074426 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.074394 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faec438d-9b43-450a-b39e-e6eabb6e0b98" path="/var/lib/kubelet/pods/faec438d-9b43-450a-b39e-e6eabb6e0b98/volumes" Oct 30 19:37:21.260028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.256455 5047 patch_prober.go:29] interesting pod/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 19:37:21.260028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.256542 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" podUID="90f0e672-c84b-4721-9a09-363ffdf05947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 19:37:21.310031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.302769 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw"] Oct 30 19:37:21.323029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.319878 5047 cpu_manager.go:414] "containerMap: removing container" podUID="faec438d-9b43-450a-b39e-e6eabb6e0b98" containerName="istio-proxy" Oct 30 19:37:21.323029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.319907 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faec438d-9b43-450a-b39e-e6eabb6e0b98" containerName="istio-proxy" Oct 30 19:37:21.323029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.320147 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="faec438d-9b43-450a-b39e-e6eabb6e0b98" containerName="istio-proxy" Oct 30 19:37:21.323029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.320168 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="faec438d-9b43-450a-b39e-e6eabb6e0b98" containerName="istio-proxy" Oct 30 19:37:21.393995 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.365461 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.514032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.513119 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw"] Oct 30 19:37:21.574414 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.574367 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmcww\" (UniqueName: \"kubernetes.io/projected/9859e88e-b269-42ac-8647-0bf70d7080cb-kube-api-access-rmcww\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.574709 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.574672 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9859e88e-b269-42ac-8647-0bf70d7080cb-istiod-ca-cert\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.575061 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.574999 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-credential-socket\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.575273 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.575241 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-envoy\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.575483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.575462 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-workload-certs\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.575863 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.575834 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-workload-socket\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.587227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.587184 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-podinfo\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.587507 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.587488 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-data\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.591028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.587725 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-token\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.747162 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.736357 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a7c3b0-openshift-default-dockercfg-gv7sh\"" Oct 30 19:37:21.887606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.871480 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-token\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.887606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.871703 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmcww\" (UniqueName: \"kubernetes.io/projected/9859e88e-b269-42ac-8647-0bf70d7080cb-kube-api-access-rmcww\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.887606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.871767 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9859e88e-b269-42ac-8647-0bf70d7080cb-istiod-ca-cert\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.887606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.871930 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-credential-socket\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.887606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.871992 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-envoy\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.887606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.872141 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-workload-certs\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.887606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.872232 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-workload-socket\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.887606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.872370 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-podinfo\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.887606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.872507 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-data\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.887606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.874953 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-data\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.887606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.877404 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9859e88e-b269-42ac-8647-0bf70d7080cb-istiod-ca-cert\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.887606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.877834 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-credential-socket\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.887606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.885808 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-workload-certs\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:21.887606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.886157 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-workload-socket\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:22.022167 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:21.956997 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:37:22.022167 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:22.008621 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-token\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:22.032814 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:22.029089 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-envoy\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:22.057622 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:22.034622 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmcww\" (UniqueName: \"kubernetes.io/projected/9859e88e-b269-42ac-8647-0bf70d7080cb-kube-api-access-rmcww\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:22.057622 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:22.057283 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-podinfo\") pod \"e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:22.139535 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:22.139495 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:22.263207 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:22.262734 5047 patch_prober.go:29] interesting pod/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 19:37:22.263207 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:22.262795 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" podUID="90f0e672-c84b-4721-9a09-363ffdf05947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 19:37:22.565706 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:22.565677 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw"] Oct 30 19:37:22.567451 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:22.567414 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:22.567566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:22.567498 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:22.567566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:22.567545 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:23.230078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:23.230032 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" event={"ID":"9859e88e-b269-42ac-8647-0bf70d7080cb","Type":"ContainerStarted","Data":"f3126db0e1961ffb3fa4ed39c1beecef6c7018289741a6b340968eebcfb38bb2"} Oct 30 19:37:23.230292 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:23.230110 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" event={"ID":"9859e88e-b269-42ac-8647-0bf70d7080cb","Type":"ContainerStarted","Data":"73d308d4a422c18be57182661036607dc59e745779836adafd757a0851137573"} Oct 30 19:37:23.260043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:23.259666 5047 patch_prober.go:29] interesting pod/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 19:37:23.260043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:23.259740 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" podUID="90f0e672-c84b-4721-9a09-363ffdf05947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 19:37:24.151791 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:24.151763 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:24.167162 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:24.166173 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:24.222660 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:24.222601 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" podStartSLOduration=3.222582087 podStartE2EDuration="3.222582087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:23.300446675 +0000 UTC m=+1996.374902031" watchObservedRunningTime="2026-10-30 19:37:24.222582087 +0000 UTC m=+1997.297037449" Oct 30 19:37:24.260880 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:24.260422 5047 patch_prober.go:29] interesting pod/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 19:37:24.260880 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:24.260491 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" podUID="90f0e672-c84b-4721-9a09-363ffdf05947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 19:37:24.266028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:24.265683 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:24.266952 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:24.266930 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:37:25.251548 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:25.251508 5047 patch_prober.go:29] interesting pod/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 19:37:25.252202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:25.252167 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" podUID="90f0e672-c84b-4721-9a09-363ffdf05947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 19:37:26.290031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:26.282130 5047 patch_prober.go:29] interesting pod/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 19:37:26.290031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:26.282197 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" podUID="90f0e672-c84b-4721-9a09-363ffdf05947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 19:37:27.265356 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:27.265322 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:37:27.271234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:27.271208 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:37:28.273101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:28.273055 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62"] Oct 30 19:37:28.273603 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:28.273321 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" podUID="43b414f8-8bde-4da5-9a34-d42552256a73" containerName="maas-api" containerID="cri-o://3aab7dd12a27c782a379595f3d383982214f38e899081c3c67d295bfd86fa55a" gracePeriod=30 Oct 30 19:37:28.958329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:28.957331 5047 generic.go:356] "Generic (PLEG): container finished" podID="43b414f8-8bde-4da5-9a34-d42552256a73" containerID="3aab7dd12a27c782a379595f3d383982214f38e899081c3c67d295bfd86fa55a" exitCode=0 Oct 30 19:37:28.958329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:28.957368 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" event={"ID":"43b414f8-8bde-4da5-9a34-d42552256a73","Type":"ContainerDied","Data":"3aab7dd12a27c782a379595f3d383982214f38e899081c3c67d295bfd86fa55a"} Oct 30 19:37:29.011512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:29.011102 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:37:29.064260 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:29.064171 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43b414f8-8bde-4da5-9a34-d42552256a73-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b414f8-8bde-4da5-9a34-d42552256a73-metrics-certs\") pod \"43b414f8-8bde-4da5-9a34-d42552256a73\" (UID: \"43b414f8-8bde-4da5-9a34-d42552256a73\") " Oct 30 19:37:29.064674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:29.064650 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b414f8-8bde-4da5-9a34-d42552256a73-kube-api-access-mqr5d\" (UniqueName: \"kubernetes.io/projected/43b414f8-8bde-4da5-9a34-d42552256a73-kube-api-access-mqr5d\") pod \"43b414f8-8bde-4da5-9a34-d42552256a73\" (UID: \"43b414f8-8bde-4da5-9a34-d42552256a73\") " Oct 30 19:37:29.064825 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:29.064812 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43b414f8-8bde-4da5-9a34-d42552256a73-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43b414f8-8bde-4da5-9a34-d42552256a73-maas-api-tls\") pod \"43b414f8-8bde-4da5-9a34-d42552256a73\" (UID: \"43b414f8-8bde-4da5-9a34-d42552256a73\") " Oct 30 19:37:29.065055 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:29.065039 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b414f8-8bde-4da5-9a34-d42552256a73-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b414f8-8bde-4da5-9a34-d42552256a73-openshift-service-ca\") pod \"43b414f8-8bde-4da5-9a34-d42552256a73\" (UID: \"43b414f8-8bde-4da5-9a34-d42552256a73\") " Oct 30 19:37:29.094811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:29.094775 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b414f8-8bde-4da5-9a34-d42552256a73-openshift-service-ca" pod "43b414f8-8bde-4da5-9a34-d42552256a73" (UID: "43b414f8-8bde-4da5-9a34-d42552256a73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:29.116302 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:29.116259 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b414f8-8bde-4da5-9a34-d42552256a73-metrics-certs" pod "43b414f8-8bde-4da5-9a34-d42552256a73" (UID: "43b414f8-8bde-4da5-9a34-d42552256a73"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:29.207029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:29.182249 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b414f8-8bde-4da5-9a34-d42552256a73-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:37:29.207029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:29.182286 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b414f8-8bde-4da5-9a34-d42552256a73-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:37:29.240945 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:29.239947 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b414f8-8bde-4da5-9a34-d42552256a73-maas-api-tls" pod "43b414f8-8bde-4da5-9a34-d42552256a73" (UID: "43b414f8-8bde-4da5-9a34-d42552256a73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:29.314046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:29.285628 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b414f8-8bde-4da5-9a34-d42552256a73-kube-api-access-mqr5d" pod "43b414f8-8bde-4da5-9a34-d42552256a73" (UID: "43b414f8-8bde-4da5-9a34-d42552256a73"). InnerVolumeSpecName "kube-api-access-mqr5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:29.410644 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:29.410603 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqr5d\" (UniqueName: \"kubernetes.io/projected/43b414f8-8bde-4da5-9a34-d42552256a73-kube-api-access-mqr5d\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:37:29.410883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:29.410862 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43b414f8-8bde-4da5-9a34-d42552256a73-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:37:29.984610 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:29.984568 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" event={"ID":"43b414f8-8bde-4da5-9a34-d42552256a73","Type":"ContainerDied","Data":"7b61af7dbfbd947c11921ec6b30d4d0d6edcece39aca480b6ca9d380f5edd2b2"} Oct 30 19:37:29.984946 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:29.984922 5047 scope.go:122] "RemoveContainer" containerID="3aab7dd12a27c782a379595f3d383982214f38e899081c3c67d295bfd86fa55a" Oct 30 19:37:29.985283 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:29.985264 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62" Oct 30 19:37:30.317124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:30.311835 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62"] Oct 30 19:37:30.500382 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:30.500339 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-092bc0-85498957b8-lcs62"] Oct 30 19:37:31.073344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:31.073307 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b414f8-8bde-4da5-9a34-d42552256a73" path="/var/lib/kubelet/pods/43b414f8-8bde-4da5-9a34-d42552256a73/volumes" Oct 30 19:37:38.105325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:38.103559 5047 patch_prober.go:29] interesting pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": read tcp 10.132.0.2:44512->10.132.0.136:15021: read: connection reset by peer" start-of-body= Oct 30 19:37:38.105325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:38.103645 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" podUID="78be6f9f-292a-4259-bb67-0d1dd1365325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": read tcp 10.132.0.2:44512->10.132.0.136:15021: read: connection reset by peer" Oct 30 19:37:39.062750 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:39.062716 5047 generic.go:356] "Generic (PLEG): container finished" podID="78be6f9f-292a-4259-bb67-0d1dd1365325" containerID="430f8a4b7e0569c133f6151c6449a8334ed5743667888b300e06850b869c9471" exitCode=0 Oct 30 19:37:39.062927 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:39.062788 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" event={"ID":"78be6f9f-292a-4259-bb67-0d1dd1365325","Type":"ContainerDied","Data":"430f8a4b7e0569c133f6151c6449a8334ed5743667888b300e06850b869c9471"} Oct 30 19:37:39.063320 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:39.063301 5047 scope.go:122] "RemoveContainer" containerID="430f8a4b7e0569c133f6151c6449a8334ed5743667888b300e06850b869c9471" Oct 30 19:37:39.064965 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:39.064937 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:39.065071 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:39.065045 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:39.065120 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:39.065083 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:39.604762 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:37:39.604723 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:37:40.073379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:40.073295 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" event={"ID":"78be6f9f-292a-4259-bb67-0d1dd1365325","Type":"ContainerStarted","Data":"aa06c200c6b227b8e85d661ede2e97b3c2d54d48f59afac57d01b5d82bcdf62f"} Oct 30 19:37:40.722768 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:40.722731 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:37:40.723292 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:40.722922 5047 patch_prober.go:29] interesting pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 19:37:40.723292 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:40.722983 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" podUID="78be6f9f-292a-4259-bb67-0d1dd1365325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 19:37:41.723294 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:41.723259 5047 patch_prober.go:29] interesting pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 19:37:41.723818 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:41.723320 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" podUID="78be6f9f-292a-4259-bb67-0d1dd1365325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 19:37:42.725501 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:42.725455 5047 patch_prober.go:29] interesting pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 19:37:42.726134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:42.726090 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" podUID="78be6f9f-292a-4259-bb67-0d1dd1365325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 19:37:43.723405 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:43.723365 5047 patch_prober.go:29] interesting pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 19:37:43.723587 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:43.723429 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" podUID="78be6f9f-292a-4259-bb67-0d1dd1365325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 19:37:44.723081 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:44.722991 5047 patch_prober.go:29] interesting pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 19:37:44.723501 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:44.723104 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" podUID="78be6f9f-292a-4259-bb67-0d1dd1365325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 19:37:45.723439 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:45.723395 5047 patch_prober.go:29] interesting pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 19:37:45.723849 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:45.723466 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" podUID="78be6f9f-292a-4259-bb67-0d1dd1365325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 19:37:46.727790 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:46.727751 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:37:46.728229 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:46.728139 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:37:46.728785 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:37:46.728768 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:38:01.927307 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:01.927265 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx"] Oct 30 19:38:01.959021 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:01.958979 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98"] Oct 30 19:38:04.967429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:04.967377 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf"] Oct 30 19:38:04.980700 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:04.980663 5047 cpu_manager.go:414] "containerMap: removing container" podUID="faec438d-9b43-450a-b39e-e6eabb6e0b98" containerName="istio-proxy" Oct 30 19:38:04.980932 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:04.980911 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faec438d-9b43-450a-b39e-e6eabb6e0b98" containerName="istio-proxy" Oct 30 19:38:04.981103 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:04.981078 5047 cpu_manager.go:414] "containerMap: removing container" podUID="43b414f8-8bde-4da5-9a34-d42552256a73" containerName="maas-api" Oct 30 19:38:04.981218 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:04.981206 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b414f8-8bde-4da5-9a34-d42552256a73" containerName="maas-api" Oct 30 19:38:04.981569 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:04.981550 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b414f8-8bde-4da5-9a34-d42552256a73" containerName="maas-api" Oct 30 19:38:05.031027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.028083 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.091435 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.091394 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf"] Oct 30 19:38:05.095119 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.095085 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9c8a2a\"/\"default-dockercfg-w6jvn\"" Oct 30 19:38:05.095295 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.095281 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9c8a2a\"/\"test-model-9c8a2a-kserve-self-signed-certs\"" Oct 30 19:38:05.095400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.095335 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9c8a2a\"/\"kube-root-ca.crt\"" Oct 30 19:38:05.095463 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.095398 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9c8a2a\"/\"openshift-service-ca.crt\"" Oct 30 19:38:05.240166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.191203 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-tmp-dir\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.240545 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.240511 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7456b22a-9c80-4fbb-9236-08eed4067902-tls-certs\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.240961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.240939 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8bzt\" (UniqueName: \"kubernetes.io/projected/7456b22a-9c80-4fbb-9236-08eed4067902-kube-api-access-d8bzt\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.241166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.241141 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-dshm\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.247150 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.247115 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-home\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.247446 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.247424 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-model-cache\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.351043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.350570 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8bzt\" (UniqueName: \"kubernetes.io/projected/7456b22a-9c80-4fbb-9236-08eed4067902-kube-api-access-d8bzt\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.351043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.350646 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-dshm\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.351043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.350676 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-home\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.351043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.350760 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-model-cache\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.351043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.350942 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-tmp-dir\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.351043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.351048 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7456b22a-9c80-4fbb-9236-08eed4067902-tls-certs\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.353035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.352992 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-home\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.353418 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.353397 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-model-cache\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.353756 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.353738 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-tmp-dir\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.356915 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.356894 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-dshm\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.364149 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.362694 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7456b22a-9c80-4fbb-9236-08eed4067902-tls-certs\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.378486 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.378403 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8bzt\" (UniqueName: \"kubernetes.io/projected/7456b22a-9c80-4fbb-9236-08eed4067902-kube-api-access-d8bzt\") pod \"test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.409816 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.409511 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:05.756918 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:38:05.756882 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7456b22a_9c80_4fbb_9236_08eed4067902.slice/crio-de3fffb7fc96b60ebddafeb226ec00468a982281333f56ecc329afcbc150e216 WatchSource:0}: Error finding container de3fffb7fc96b60ebddafeb226ec00468a982281333f56ecc329afcbc150e216: Status 404 returned error can't find the container with id de3fffb7fc96b60ebddafeb226ec00468a982281333f56ecc329afcbc150e216 Oct 30 19:38:05.758830 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:05.758802 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf"] Oct 30 19:38:06.845748 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:06.845651 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" event={"ID":"7456b22a-9c80-4fbb-9236-08eed4067902","Type":"ContainerStarted","Data":"c8ad6e743a0419c65ea38f517284ca2c155a5070c5ab8d0c8824502908afc6de"} Oct 30 19:38:06.845748 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:06.845714 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" event={"ID":"7456b22a-9c80-4fbb-9236-08eed4067902","Type":"ContainerStarted","Data":"de3fffb7fc96b60ebddafeb226ec00468a982281333f56ecc329afcbc150e216"} Oct 30 19:38:06.885595 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:06.885534 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" podStartSLOduration=2.8855195829999998 podStartE2EDuration="2.885519583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:06.884191933 +0000 UTC m=+2039.958647292" watchObservedRunningTime="2026-10-30 19:38:06.885519583 +0000 UTC m=+2039.959974941" Oct 30 19:38:06.927954 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:06.927916 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" podUID="cc1de489-6ed4-4299-9af0-e22c3f80dbb3" containerName="payload-processing" containerID="cri-o://28cb4b24386d03e39bb5a4af7885c6ffe0d57cd4cf07c07e40099b8b57c35243" gracePeriod=25 Oct 30 19:38:06.959949 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:06.959912 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" podUID="b9deac3a-adad-4f75-8c98-d4fdeeb5b223" containerName="payload-pre-processing" containerID="cri-o://ffc94697db48c0bbf3a760bd230c5b41a9f0d3202f87308895904b01090c9cf0" gracePeriod=25 Oct 30 19:38:07.355285 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.355257 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" Oct 30 19:38:07.360996 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.360933 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" Oct 30 19:38:07.411090 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.411058 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:07.493542 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.493508 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1de489-6ed4-4299-9af0-e22c3f80dbb3-kube-api-access-hvkpt\" (UniqueName: \"kubernetes.io/projected/cc1de489-6ed4-4299-9af0-e22c3f80dbb3-kube-api-access-hvkpt\") pod \"cc1de489-6ed4-4299-9af0-e22c3f80dbb3\" (UID: \"cc1de489-6ed4-4299-9af0-e22c3f80dbb3\") " Oct 30 19:38:07.493752 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.493605 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1de489-6ed4-4299-9af0-e22c3f80dbb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc1de489-6ed4-4299-9af0-e22c3f80dbb3-plugins-config-volume\") pod \"cc1de489-6ed4-4299-9af0-e22c3f80dbb3\" (UID: \"cc1de489-6ed4-4299-9af0-e22c3f80dbb3\") " Oct 30 19:38:07.493752 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.493732 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9deac3a-adad-4f75-8c98-d4fdeeb5b223-kube-api-access-6kvsq\" (UniqueName: \"kubernetes.io/projected/b9deac3a-adad-4f75-8c98-d4fdeeb5b223-kube-api-access-6kvsq\") pod \"b9deac3a-adad-4f75-8c98-d4fdeeb5b223\" (UID: \"b9deac3a-adad-4f75-8c98-d4fdeeb5b223\") " Oct 30 19:38:07.493868 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.493844 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9deac3a-adad-4f75-8c98-d4fdeeb5b223-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9deac3a-adad-4f75-8c98-d4fdeeb5b223-plugins-config-volume\") pod \"b9deac3a-adad-4f75-8c98-d4fdeeb5b223\" (UID: \"b9deac3a-adad-4f75-8c98-d4fdeeb5b223\") " Oct 30 19:38:07.494111 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.494081 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1de489-6ed4-4299-9af0-e22c3f80dbb3-plugins-config-volume" pod "cc1de489-6ed4-4299-9af0-e22c3f80dbb3" (UID: "cc1de489-6ed4-4299-9af0-e22c3f80dbb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:07.494360 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.494308 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9deac3a-adad-4f75-8c98-d4fdeeb5b223-plugins-config-volume" pod "b9deac3a-adad-4f75-8c98-d4fdeeb5b223" (UID: "b9deac3a-adad-4f75-8c98-d4fdeeb5b223"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:07.495415 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.495385 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc1de489-6ed4-4299-9af0-e22c3f80dbb3-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:07.495527 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.495418 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9deac3a-adad-4f75-8c98-d4fdeeb5b223-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:07.495845 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.495823 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1de489-6ed4-4299-9af0-e22c3f80dbb3-kube-api-access-hvkpt" pod "cc1de489-6ed4-4299-9af0-e22c3f80dbb3" (UID: "cc1de489-6ed4-4299-9af0-e22c3f80dbb3"). InnerVolumeSpecName "kube-api-access-hvkpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:07.496344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.496319 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9deac3a-adad-4f75-8c98-d4fdeeb5b223-kube-api-access-6kvsq" pod "b9deac3a-adad-4f75-8c98-d4fdeeb5b223" (UID: "b9deac3a-adad-4f75-8c98-d4fdeeb5b223"). InnerVolumeSpecName "kube-api-access-6kvsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:07.596933 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.596900 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kvsq\" (UniqueName: \"kubernetes.io/projected/b9deac3a-adad-4f75-8c98-d4fdeeb5b223-kube-api-access-6kvsq\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:07.596933 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.596932 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvkpt\" (UniqueName: \"kubernetes.io/projected/cc1de489-6ed4-4299-9af0-e22c3f80dbb3-kube-api-access-hvkpt\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:07.867737 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.867700 5047 generic.go:356] "Generic (PLEG): container finished" podID="cc1de489-6ed4-4299-9af0-e22c3f80dbb3" containerID="28cb4b24386d03e39bb5a4af7885c6ffe0d57cd4cf07c07e40099b8b57c35243" exitCode=0 Oct 30 19:38:07.868258 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.867765 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" Oct 30 19:38:07.868258 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.867776 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" event={"ID":"cc1de489-6ed4-4299-9af0-e22c3f80dbb3","Type":"ContainerDied","Data":"28cb4b24386d03e39bb5a4af7885c6ffe0d57cd4cf07c07e40099b8b57c35243"} Oct 30 19:38:07.868258 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.867859 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx" event={"ID":"cc1de489-6ed4-4299-9af0-e22c3f80dbb3","Type":"ContainerDied","Data":"25bdb3859c39c44055e09fc059b61b7c8473d65d89f681e051519921904f9653"} Oct 30 19:38:07.868258 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.867874 5047 scope.go:122] "RemoveContainer" containerID="28cb4b24386d03e39bb5a4af7885c6ffe0d57cd4cf07c07e40099b8b57c35243" Oct 30 19:38:07.869536 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.869511 5047 generic.go:356] "Generic (PLEG): container finished" podID="b9deac3a-adad-4f75-8c98-d4fdeeb5b223" containerID="ffc94697db48c0bbf3a760bd230c5b41a9f0d3202f87308895904b01090c9cf0" exitCode=0 Oct 30 19:38:07.869760 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.869732 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" event={"ID":"b9deac3a-adad-4f75-8c98-d4fdeeb5b223","Type":"ContainerDied","Data":"ffc94697db48c0bbf3a760bd230c5b41a9f0d3202f87308895904b01090c9cf0"} Oct 30 19:38:07.869867 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.869785 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" event={"ID":"b9deac3a-adad-4f75-8c98-d4fdeeb5b223","Type":"ContainerDied","Data":"3ac5f0c7b68cf1f925d69a50c109b713ec4f6ee810d564511a8b118bf326a554"} Oct 30 19:38:07.869867 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.869831 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98" Oct 30 19:38:07.880063 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.879915 5047 scope.go:122] "RemoveContainer" containerID="28cb4b24386d03e39bb5a4af7885c6ffe0d57cd4cf07c07e40099b8b57c35243" Oct 30 19:38:07.880327 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:38:07.880294 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28cb4b24386d03e39bb5a4af7885c6ffe0d57cd4cf07c07e40099b8b57c35243\": container with ID starting with 28cb4b24386d03e39bb5a4af7885c6ffe0d57cd4cf07c07e40099b8b57c35243 not found: ID does not exist" containerID="28cb4b24386d03e39bb5a4af7885c6ffe0d57cd4cf07c07e40099b8b57c35243" Oct 30 19:38:07.880445 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.880335 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28cb4b24386d03e39bb5a4af7885c6ffe0d57cd4cf07c07e40099b8b57c35243"} err="failed to get container status \"28cb4b24386d03e39bb5a4af7885c6ffe0d57cd4cf07c07e40099b8b57c35243\": rpc error: code = NotFound desc = could not find container \"28cb4b24386d03e39bb5a4af7885c6ffe0d57cd4cf07c07e40099b8b57c35243\": container with ID starting with 28cb4b24386d03e39bb5a4af7885c6ffe0d57cd4cf07c07e40099b8b57c35243 not found: ID does not exist" Oct 30 19:38:07.880445 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.880361 5047 scope.go:122] "RemoveContainer" containerID="ffc94697db48c0bbf3a760bd230c5b41a9f0d3202f87308895904b01090c9cf0" Oct 30 19:38:07.891581 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.891556 5047 scope.go:122] "RemoveContainer" containerID="ffc94697db48c0bbf3a760bd230c5b41a9f0d3202f87308895904b01090c9cf0" Oct 30 19:38:07.891886 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:38:07.891860 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffc94697db48c0bbf3a760bd230c5b41a9f0d3202f87308895904b01090c9cf0\": container with ID starting with ffc94697db48c0bbf3a760bd230c5b41a9f0d3202f87308895904b01090c9cf0 not found: ID does not exist" containerID="ffc94697db48c0bbf3a760bd230c5b41a9f0d3202f87308895904b01090c9cf0" Oct 30 19:38:07.891993 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.891894 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffc94697db48c0bbf3a760bd230c5b41a9f0d3202f87308895904b01090c9cf0"} err="failed to get container status \"ffc94697db48c0bbf3a760bd230c5b41a9f0d3202f87308895904b01090c9cf0\": rpc error: code = NotFound desc = could not find container \"ffc94697db48c0bbf3a760bd230c5b41a9f0d3202f87308895904b01090c9cf0\": container with ID starting with ffc94697db48c0bbf3a760bd230c5b41a9f0d3202f87308895904b01090c9cf0 not found: ID does not exist" Oct 30 19:38:07.933104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.933071 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx"] Oct 30 19:38:07.944262 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.944232 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ccf8bd61-78478dd5bb-j5qhx"] Oct 30 19:38:07.971085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.970041 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98"] Oct 30 19:38:07.974660 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:07.974636 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ccf8bd61-749b86b7ff-jdx98"] Oct 30 19:38:09.055192 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:09.055155 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9deac3a-adad-4f75-8c98-d4fdeeb5b223" path="/var/lib/kubelet/pods/b9deac3a-adad-4f75-8c98-d4fdeeb5b223/volumes" Oct 30 19:38:09.055703 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:09.055688 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc1de489-6ed4-4299-9af0-e22c3f80dbb3" path="/var/lib/kubelet/pods/cc1de489-6ed4-4299-9af0-e22c3f80dbb3/volumes" Oct 30 19:38:09.208318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:09.208284 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm"] Oct 30 19:38:09.604798 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:38:09.604759 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:38:13.394505 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:13.394467 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd"] Oct 30 19:38:13.394971 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:13.394724 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" podUID="edca9e01-ae45-430c-8478-07d1667a7c07" containerName="maas-api" containerID="cri-o://5bbcb82b1db33924aca6539b364b150383b695559abf7eacb616bafd00a3a674" gracePeriod=30 Oct 30 19:38:14.015181 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.013983 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:38:14.035657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.035338 5047 generic.go:356] "Generic (PLEG): container finished" podID="edca9e01-ae45-430c-8478-07d1667a7c07" containerID="5bbcb82b1db33924aca6539b364b150383b695559abf7eacb616bafd00a3a674" exitCode=0 Oct 30 19:38:14.037206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.036689 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" event={"ID":"edca9e01-ae45-430c-8478-07d1667a7c07","Type":"ContainerDied","Data":"5bbcb82b1db33924aca6539b364b150383b695559abf7eacb616bafd00a3a674"} Oct 30 19:38:14.037206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.036757 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" event={"ID":"edca9e01-ae45-430c-8478-07d1667a7c07","Type":"ContainerDied","Data":"d227716ad5aaeca4e850ee4d974606ad864e647ca48e5a9881d8909d03c3348e"} Oct 30 19:38:14.037206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.036789 5047 scope.go:122] "RemoveContainer" containerID="5bbcb82b1db33924aca6539b364b150383b695559abf7eacb616bafd00a3a674" Oct 30 19:38:14.037206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.037047 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd" Oct 30 19:38:14.086947 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.086898 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edca9e01-ae45-430c-8478-07d1667a7c07-metrics-certs\" (UniqueName: \"kubernetes.io/secret/edca9e01-ae45-430c-8478-07d1667a7c07-metrics-certs\") pod \"edca9e01-ae45-430c-8478-07d1667a7c07\" (UID: \"edca9e01-ae45-430c-8478-07d1667a7c07\") " Oct 30 19:38:14.087318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.087274 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edca9e01-ae45-430c-8478-07d1667a7c07-kube-api-access-l2cqv\" (UniqueName: \"kubernetes.io/projected/edca9e01-ae45-430c-8478-07d1667a7c07-kube-api-access-l2cqv\") pod \"edca9e01-ae45-430c-8478-07d1667a7c07\" (UID: \"edca9e01-ae45-430c-8478-07d1667a7c07\") " Oct 30 19:38:14.087433 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.087371 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edca9e01-ae45-430c-8478-07d1667a7c07-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edca9e01-ae45-430c-8478-07d1667a7c07-openshift-service-ca\") pod \"edca9e01-ae45-430c-8478-07d1667a7c07\" (UID: \"edca9e01-ae45-430c-8478-07d1667a7c07\") " Oct 30 19:38:14.087712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.087686 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edca9e01-ae45-430c-8478-07d1667a7c07-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edca9e01-ae45-430c-8478-07d1667a7c07-maas-api-tls\") pod \"edca9e01-ae45-430c-8478-07d1667a7c07\" (UID: \"edca9e01-ae45-430c-8478-07d1667a7c07\") " Oct 30 19:38:14.095501 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.095465 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edca9e01-ae45-430c-8478-07d1667a7c07-metrics-certs" pod "edca9e01-ae45-430c-8478-07d1667a7c07" (UID: "edca9e01-ae45-430c-8478-07d1667a7c07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:14.095696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.095651 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edca9e01-ae45-430c-8478-07d1667a7c07-openshift-service-ca" pod "edca9e01-ae45-430c-8478-07d1667a7c07" (UID: "edca9e01-ae45-430c-8478-07d1667a7c07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:14.097981 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.097925 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edca9e01-ae45-430c-8478-07d1667a7c07-maas-api-tls" pod "edca9e01-ae45-430c-8478-07d1667a7c07" (UID: "edca9e01-ae45-430c-8478-07d1667a7c07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:14.104097 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.103562 5047 scope.go:122] "RemoveContainer" containerID="5bbcb82b1db33924aca6539b364b150383b695559abf7eacb616bafd00a3a674" Oct 30 19:38:14.107938 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:38:14.107902 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bbcb82b1db33924aca6539b364b150383b695559abf7eacb616bafd00a3a674\": container with ID starting with 5bbcb82b1db33924aca6539b364b150383b695559abf7eacb616bafd00a3a674 not found: ID does not exist" containerID="5bbcb82b1db33924aca6539b364b150383b695559abf7eacb616bafd00a3a674" Oct 30 19:38:14.108301 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.108271 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bbcb82b1db33924aca6539b364b150383b695559abf7eacb616bafd00a3a674"} err="failed to get container status \"5bbcb82b1db33924aca6539b364b150383b695559abf7eacb616bafd00a3a674\": rpc error: code = NotFound desc = could not find container \"5bbcb82b1db33924aca6539b364b150383b695559abf7eacb616bafd00a3a674\": container with ID starting with 5bbcb82b1db33924aca6539b364b150383b695559abf7eacb616bafd00a3a674 not found: ID does not exist" Oct 30 19:38:14.108698 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.108670 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edca9e01-ae45-430c-8478-07d1667a7c07-kube-api-access-l2cqv" pod "edca9e01-ae45-430c-8478-07d1667a7c07" (UID: "edca9e01-ae45-430c-8478-07d1667a7c07"). InnerVolumeSpecName "kube-api-access-l2cqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:14.203215 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.197207 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edca9e01-ae45-430c-8478-07d1667a7c07-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:14.203474 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.203451 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edca9e01-ae45-430c-8478-07d1667a7c07-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:14.203598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.203584 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2cqv\" (UniqueName: \"kubernetes.io/projected/edca9e01-ae45-430c-8478-07d1667a7c07-kube-api-access-l2cqv\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:14.217128 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.217076 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edca9e01-ae45-430c-8478-07d1667a7c07-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:14.476192 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.476102 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd"] Oct 30 19:38:14.482456 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:14.482424 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7d9e82-d64994d4-z5ktd"] Oct 30 19:38:15.064793 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:15.064755 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edca9e01-ae45-430c-8478-07d1667a7c07" path="/var/lib/kubelet/pods/edca9e01-ae45-430c-8478-07d1667a7c07/volumes" Oct 30 19:38:15.410646 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:15.410608 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:15.424494 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:15.424465 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:16.080977 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:16.080949 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:38:20.941637 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.941595 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr"] Oct 30 19:38:20.942861 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.942833 5047 cpu_manager.go:414] "containerMap: removing container" podUID="cc1de489-6ed4-4299-9af0-e22c3f80dbb3" containerName="payload-processing" Oct 30 19:38:20.943000 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.942958 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1de489-6ed4-4299-9af0-e22c3f80dbb3" containerName="payload-processing" Oct 30 19:38:20.943118 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.943102 5047 cpu_manager.go:414] "containerMap: removing container" podUID="b9deac3a-adad-4f75-8c98-d4fdeeb5b223" containerName="payload-pre-processing" Oct 30 19:38:20.943178 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.943117 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9deac3a-adad-4f75-8c98-d4fdeeb5b223" containerName="payload-pre-processing" Oct 30 19:38:20.943178 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.943162 5047 cpu_manager.go:414] "containerMap: removing container" podUID="edca9e01-ae45-430c-8478-07d1667a7c07" containerName="maas-api" Oct 30 19:38:20.943178 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.943170 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edca9e01-ae45-430c-8478-07d1667a7c07" containerName="maas-api" Oct 30 19:38:20.943411 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.943391 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9deac3a-adad-4f75-8c98-d4fdeeb5b223" containerName="payload-pre-processing" Oct 30 19:38:20.943472 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.943418 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="edca9e01-ae45-430c-8478-07d1667a7c07" containerName="maas-api" Oct 30 19:38:20.943472 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.943439 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1de489-6ed4-4299-9af0-e22c3f80dbb3" containerName="payload-processing" Oct 30 19:38:20.947689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.947666 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:20.965379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.958906 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-workload-certs\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:20.965379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.959157 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-envoy\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:20.965379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.959223 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t94z5\" (UniqueName: \"kubernetes.io/projected/c4296edd-bfda-4c0d-898a-55bb6c711682-kube-api-access-t94z5\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:20.965379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.959324 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-workload-socket\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:20.965379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.959370 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-data\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:20.965379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.959407 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-credential-socket\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:20.965379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.959435 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-token\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:20.965379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.959462 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4296edd-bfda-4c0d-898a-55bb6c711682-istiod-ca-cert\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:20.965379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.959510 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-podinfo\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:20.980295 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.980264 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5d92fa-openshift-default-dockercfg-9z4jq\"" Oct 30 19:38:20.987147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:20.987116 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr"] Oct 30 19:38:21.060603 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.060570 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-envoy\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.060785 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.060629 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t94z5\" (UniqueName: \"kubernetes.io/projected/c4296edd-bfda-4c0d-898a-55bb6c711682-kube-api-access-t94z5\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.060785 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.060728 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-workload-socket\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.060785 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.060771 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-data\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.060961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.060805 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-credential-socket\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.060961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.060834 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-token\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.060961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.060858 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4296edd-bfda-4c0d-898a-55bb6c711682-istiod-ca-cert\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.060961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.060903 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-podinfo\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.061181 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.061076 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-workload-certs\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.061262 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.061231 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-workload-socket\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.061562 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.061540 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-data\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.061744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.061722 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-workload-certs\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.061833 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.061810 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-credential-socket\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.061894 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.061828 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4296edd-bfda-4c0d-898a-55bb6c711682-istiod-ca-cert\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.063329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.063307 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-envoy\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.063858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.063841 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-podinfo\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.081553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.081524 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t94z5\" (UniqueName: \"kubernetes.io/projected/c4296edd-bfda-4c0d-898a-55bb6c711682-kube-api-access-t94z5\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.082041 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.082018 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-token\") pod \"e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.266412 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.266320 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:21.860982 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:38:21.860954 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4296edd_bfda_4c0d_898a_55bb6c711682.slice/crio-b8c829d8f02057870b3fe9e89c4a738c1d5b2775720c5b936a3a31619b8de8ab WatchSource:0}: Error finding container b8c829d8f02057870b3fe9e89c4a738c1d5b2775720c5b936a3a31619b8de8ab: Status 404 returned error can't find the container with id b8c829d8f02057870b3fe9e89c4a738c1d5b2775720c5b936a3a31619b8de8ab Oct 30 19:38:21.862371 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.862345 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr"] Oct 30 19:38:21.863998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.863964 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:38:21.864091 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.864050 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:38:21.864091 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:21.864082 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:38:22.169350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:22.168841 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" event={"ID":"c4296edd-bfda-4c0d-898a-55bb6c711682","Type":"ContainerStarted","Data":"6463058d753528fe240ad999bc35d4bad4f45e6f06fbbf55e9206ad0c8ad1956"} Oct 30 19:38:22.169350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:22.168917 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" event={"ID":"c4296edd-bfda-4c0d-898a-55bb6c711682","Type":"ContainerStarted","Data":"b8c829d8f02057870b3fe9e89c4a738c1d5b2775720c5b936a3a31619b8de8ab"} Oct 30 19:38:22.213998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:22.213942 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" podStartSLOduration=2.213925841 podStartE2EDuration="2.213925841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:22.211915879 +0000 UTC m=+2055.286371242" watchObservedRunningTime="2026-10-30 19:38:22.213925841 +0000 UTC m=+2055.288381198" Oct 30 19:38:22.267665 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:22.267632 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:22.269076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:22.269050 5047 patch_prober.go:29] interesting pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:38:22.269205 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:22.269097 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" podUID="c4296edd-bfda-4c0d-898a-55bb6c711682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 19:38:22.885416 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:22.885376 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2"] Oct 30 19:38:22.885720 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:22.885647 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" podUID="5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" containerName="istio-proxy" containerID="cri-o://9dba553322c262ad1953012029f6be119177a7b2b7482c6aaabe1a1d4c8e8103" gracePeriod=30 Oct 30 19:38:23.313812 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:23.313349 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:24.231914 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.231878 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" podUID="c935589c-ed56-4012-85fd-b628a0e50980" containerName="main" containerID="cri-o://8ec4f77584ac576d026a77e8d6a8ca3b0e6764d09b58187433bafb517529680a" gracePeriod=45 Oct 30 19:38:24.235013 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.234971 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:24.235979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.235953 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:38:24.566875 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.566844 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:38:24.726528 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.726493 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-tmp-dir\") pod \"c935589c-ed56-4012-85fd-b628a0e50980\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " Oct 30 19:38:24.726726 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.726615 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-dshm\") pod \"c935589c-ed56-4012-85fd-b628a0e50980\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " Oct 30 19:38:24.726726 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.726646 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-model-cache\") pod \"c935589c-ed56-4012-85fd-b628a0e50980\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " Oct 30 19:38:24.726846 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.726783 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-tmp-dir" pod "c935589c-ed56-4012-85fd-b628a0e50980" (UID: "c935589c-ed56-4012-85fd-b628a0e50980"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:24.726902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.726844 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c935589c-ed56-4012-85fd-b628a0e50980-tls-certs\" (UniqueName: \"kubernetes.io/secret/c935589c-ed56-4012-85fd-b628a0e50980-tls-certs\") pod \"c935589c-ed56-4012-85fd-b628a0e50980\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " Oct 30 19:38:24.726902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.726888 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c935589c-ed56-4012-85fd-b628a0e50980-kube-api-access-8hzxr\" (UniqueName: \"kubernetes.io/projected/c935589c-ed56-4012-85fd-b628a0e50980-kube-api-access-8hzxr\") pod \"c935589c-ed56-4012-85fd-b628a0e50980\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " Oct 30 19:38:24.727022 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.726978 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-model-cache" pod "c935589c-ed56-4012-85fd-b628a0e50980" (UID: "c935589c-ed56-4012-85fd-b628a0e50980"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:24.727085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.727048 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-home\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-home\") pod \"c935589c-ed56-4012-85fd-b628a0e50980\" (UID: \"c935589c-ed56-4012-85fd-b628a0e50980\") " Oct 30 19:38:24.727331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.727302 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-home" pod "c935589c-ed56-4012-85fd-b628a0e50980" (UID: "c935589c-ed56-4012-85fd-b628a0e50980"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:24.727756 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.727735 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:24.727849 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.727761 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:24.727849 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.727774 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:24.729119 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.729099 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c935589c-ed56-4012-85fd-b628a0e50980-tls-certs" pod "c935589c-ed56-4012-85fd-b628a0e50980" (UID: "c935589c-ed56-4012-85fd-b628a0e50980"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:24.729327 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.729304 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c935589c-ed56-4012-85fd-b628a0e50980-kube-api-access-8hzxr" pod "c935589c-ed56-4012-85fd-b628a0e50980" (UID: "c935589c-ed56-4012-85fd-b628a0e50980"). InnerVolumeSpecName "kube-api-access-8hzxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:24.729404 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.729388 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-dshm" pod "c935589c-ed56-4012-85fd-b628a0e50980" (UID: "c935589c-ed56-4012-85fd-b628a0e50980"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:24.829604 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.829578 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c935589c-ed56-4012-85fd-b628a0e50980-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:24.829604 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.829604 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c935589c-ed56-4012-85fd-b628a0e50980-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:24.829809 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:24.829615 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hzxr\" (UniqueName: \"kubernetes.io/projected/c935589c-ed56-4012-85fd-b628a0e50980-kube-api-access-8hzxr\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:25.243274 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:25.243152 5047 generic.go:356] "Generic (PLEG): container finished" podID="c935589c-ed56-4012-85fd-b628a0e50980" containerID="8ec4f77584ac576d026a77e8d6a8ca3b0e6764d09b58187433bafb517529680a" exitCode=0 Oct 30 19:38:25.243274 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:25.243194 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" Oct 30 19:38:25.243274 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:25.243243 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" event={"ID":"c935589c-ed56-4012-85fd-b628a0e50980","Type":"ContainerDied","Data":"8ec4f77584ac576d026a77e8d6a8ca3b0e6764d09b58187433bafb517529680a"} Oct 30 19:38:25.243630 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:25.243292 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" event={"ID":"c935589c-ed56-4012-85fd-b628a0e50980","Type":"ContainerDied","Data":"f3176fdc853345bf35cf46374290676e44765b692ef74d8f16f643e7d52b2408"} Oct 30 19:38:25.243630 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:25.243315 5047 scope.go:122] "RemoveContainer" containerID="8ec4f77584ac576d026a77e8d6a8ca3b0e6764d09b58187433bafb517529680a" Oct 30 19:38:25.257622 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:25.257586 5047 scope.go:122] "RemoveContainer" containerID="8ec4f77584ac576d026a77e8d6a8ca3b0e6764d09b58187433bafb517529680a" Oct 30 19:38:25.258037 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:38:25.257976 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec4f77584ac576d026a77e8d6a8ca3b0e6764d09b58187433bafb517529680a\": container with ID starting with 8ec4f77584ac576d026a77e8d6a8ca3b0e6764d09b58187433bafb517529680a not found: ID does not exist" containerID="8ec4f77584ac576d026a77e8d6a8ca3b0e6764d09b58187433bafb517529680a" Oct 30 19:38:25.258104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:25.258039 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec4f77584ac576d026a77e8d6a8ca3b0e6764d09b58187433bafb517529680a"} err="failed to get container status \"8ec4f77584ac576d026a77e8d6a8ca3b0e6764d09b58187433bafb517529680a\": rpc error: code = NotFound desc = could not find container \"8ec4f77584ac576d026a77e8d6a8ca3b0e6764d09b58187433bafb517529680a\": container with ID starting with 8ec4f77584ac576d026a77e8d6a8ca3b0e6764d09b58187433bafb517529680a not found: ID does not exist" Oct 30 19:38:25.298373 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:25.298339 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm"] Oct 30 19:38:25.309960 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:25.309923 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm"] Oct 30 19:38:25.477750 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:25.477706 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-ccf8bd61/e2e-lifecycle-model-ccf8bd61-kserve-848d45cfd9-69vhm" podUID="c935589c-ed56-4012-85fd-b628a0e50980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:38:27.119386 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:27.119348 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c935589c-ed56-4012-85fd-b628a0e50980" path="/var/lib/kubelet/pods/c935589c-ed56-4012-85fd-b628a0e50980/volumes" Oct 30 19:38:28.272066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.270256 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:38:28.296030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.288970 5047 generic.go:356] "Generic (PLEG): container finished" podID="5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" containerID="9dba553322c262ad1953012029f6be119177a7b2b7482c6aaabe1a1d4c8e8103" exitCode=0 Oct 30 19:38:28.296030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.289314 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" event={"ID":"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96","Type":"ContainerDied","Data":"9dba553322c262ad1953012029f6be119177a7b2b7482c6aaabe1a1d4c8e8103"} Oct 30 19:38:28.296030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.289374 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" event={"ID":"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96","Type":"ContainerDied","Data":"061bcb80d1fe10fdc4a7ab1787ec9b92aaa156ea67371696402058349170667e"} Oct 30 19:38:28.296030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.289406 5047 scope.go:122] "RemoveContainer" containerID="9dba553322c262ad1953012029f6be119177a7b2b7482c6aaabe1a1d4c8e8103" Oct 30 19:38:28.296030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.289574 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2" Oct 30 19:38:28.326196 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.323526 5047 scope.go:122] "RemoveContainer" containerID="ee8d8c04f7a511e00231382c4cbba30d2e7ff086f1260edcd3dfe2016dc256db" Oct 30 19:38:28.392826 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.392803 5047 scope.go:122] "RemoveContainer" containerID="9dba553322c262ad1953012029f6be119177a7b2b7482c6aaabe1a1d4c8e8103" Oct 30 19:38:28.393194 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:38:28.393173 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dba553322c262ad1953012029f6be119177a7b2b7482c6aaabe1a1d4c8e8103\": container with ID starting with 9dba553322c262ad1953012029f6be119177a7b2b7482c6aaabe1a1d4c8e8103 not found: ID does not exist" containerID="9dba553322c262ad1953012029f6be119177a7b2b7482c6aaabe1a1d4c8e8103" Oct 30 19:38:28.393298 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.393207 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dba553322c262ad1953012029f6be119177a7b2b7482c6aaabe1a1d4c8e8103"} err="failed to get container status \"9dba553322c262ad1953012029f6be119177a7b2b7482c6aaabe1a1d4c8e8103\": rpc error: code = NotFound desc = could not find container \"9dba553322c262ad1953012029f6be119177a7b2b7482c6aaabe1a1d4c8e8103\": container with ID starting with 9dba553322c262ad1953012029f6be119177a7b2b7482c6aaabe1a1d4c8e8103 not found: ID does not exist" Oct 30 19:38:28.393298 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.393231 5047 scope.go:122] "RemoveContainer" containerID="ee8d8c04f7a511e00231382c4cbba30d2e7ff086f1260edcd3dfe2016dc256db" Oct 30 19:38:28.393488 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:38:28.393469 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee8d8c04f7a511e00231382c4cbba30d2e7ff086f1260edcd3dfe2016dc256db\": container with ID starting with ee8d8c04f7a511e00231382c4cbba30d2e7ff086f1260edcd3dfe2016dc256db not found: ID does not exist" containerID="ee8d8c04f7a511e00231382c4cbba30d2e7ff086f1260edcd3dfe2016dc256db" Oct 30 19:38:28.393564 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.393495 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee8d8c04f7a511e00231382c4cbba30d2e7ff086f1260edcd3dfe2016dc256db"} err="failed to get container status \"ee8d8c04f7a511e00231382c4cbba30d2e7ff086f1260edcd3dfe2016dc256db\": rpc error: code = NotFound desc = could not find container \"ee8d8c04f7a511e00231382c4cbba30d2e7ff086f1260edcd3dfe2016dc256db\": container with ID starting with ee8d8c04f7a511e00231382c4cbba30d2e7ff086f1260edcd3dfe2016dc256db not found: ID does not exist" Oct 30 19:38:28.403900 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.403875 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-kube-api-access-sdkp7\" (UniqueName: \"kubernetes.io/projected/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-kube-api-access-sdkp7\") pod \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " Oct 30 19:38:28.404063 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.403929 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istiod-ca-cert\") pod \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " Oct 30 19:38:28.404063 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.403980 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-workload-certs\") pod \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " Oct 30 19:38:28.404063 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.404017 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-token\" (UniqueName: \"kubernetes.io/projected/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-token\") pod \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " Oct 30 19:38:28.404063 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.404040 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-envoy\") pod \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " Oct 30 19:38:28.404299 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.404139 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-credential-socket\") pod \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " Oct 30 19:38:28.404299 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.404159 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-podinfo\") pod \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " Oct 30 19:38:28.404299 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.404219 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-data\") pod \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " Oct 30 19:38:28.404299 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.404270 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-workload-socket\") pod \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\" (UID: \"5b488a6e-9b1c-44c0-8656-0bd32d3e3b96\") " Oct 30 19:38:28.404511 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.404288 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-workload-certs" pod "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" (UID: "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:28.404511 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.404306 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istiod-ca-cert" pod "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" (UID: "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:28.404616 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.404537 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-workload-socket" pod "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" (UID: "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:28.405448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.404699 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-data" pod "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" (UID: "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:28.405448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.404890 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-credential-socket" pod "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" (UID: "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:28.405448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.405362 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:28.405448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.405383 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:28.405448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.405397 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:28.405448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.405411 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:28.405448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.405426 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:28.406917 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.406866 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-podinfo" pod "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" (UID: "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:38:28.407145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.407105 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-envoy" pod "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" (UID: "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:28.407211 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.407150 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-kube-api-access-sdkp7" pod "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" (UID: "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96"). InnerVolumeSpecName "kube-api-access-sdkp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:28.407211 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.407179 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-token" pod "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" (UID: "5b488a6e-9b1c-44c0-8656-0bd32d3e3b96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:28.507582 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.507495 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdkp7\" (UniqueName: \"kubernetes.io/projected/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-kube-api-access-sdkp7\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:28.507582 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.507532 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:28.507582 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.507541 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:28.507582 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.507551 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:28.637697 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.637654 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2"] Oct 30 19:38:28.641718 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:28.641686 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7d9e82-openshift-default-5d58d7498f-7psm2"] Oct 30 19:38:29.051019 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:29.050969 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" path="/var/lib/kubelet/pods/5b488a6e-9b1c-44c0-8656-0bd32d3e3b96/volumes" Oct 30 19:38:34.267977 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.267935 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h"] Oct 30 19:38:34.269175 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.269153 5047 cpu_manager.go:414] "containerMap: removing container" podUID="5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" containerName="istio-proxy" Oct 30 19:38:34.269280 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.269177 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" containerName="istio-proxy" Oct 30 19:38:34.269280 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.269192 5047 cpu_manager.go:414] "containerMap: removing container" podUID="c935589c-ed56-4012-85fd-b628a0e50980" containerName="main" Oct 30 19:38:34.269280 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.269200 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c935589c-ed56-4012-85fd-b628a0e50980" containerName="main" Oct 30 19:38:34.269450 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.269433 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" containerName="istio-proxy" Oct 30 19:38:34.269450 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.269445 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="c935589c-ed56-4012-85fd-b628a0e50980" containerName="main" Oct 30 19:38:34.269551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.269457 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" containerName="istio-proxy" Oct 30 19:38:34.274024 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.273985 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.292993 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.292962 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h"] Oct 30 19:38:34.298403 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.298379 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-08bcfe-openshift-default-dockercfg-hdsw9\"" Oct 30 19:38:34.535082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.520484 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-credential-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.535082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.520632 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-envoy\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.535082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.520698 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-podinfo\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.535082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.520808 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-workload-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.535082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.520848 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-workload-certs\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.535082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.520889 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istiod-ca-cert\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.535082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.520923 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-data\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.535082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.520952 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rcfn\" (UniqueName: \"kubernetes.io/projected/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-kube-api-access-2rcfn\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.535082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.520997 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-token\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.634365 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.625446 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-envoy\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.634365 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.625538 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-podinfo\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.634365 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.625661 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-workload-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.634365 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.625710 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-workload-certs\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.634365 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.625748 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istiod-ca-cert\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.634365 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.625793 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-data\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.634365 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.625822 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rcfn\" (UniqueName: \"kubernetes.io/projected/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-kube-api-access-2rcfn\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.634365 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.625883 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-token\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.634365 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.626261 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-credential-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.634365 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.627049 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-credential-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.634365 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.628282 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istiod-ca-cert\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.635580 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.635499 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-data\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.636833 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.636081 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-workload-socket\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.636833 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.636463 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-workload-certs\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.661310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.659638 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-podinfo\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.693082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.685610 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-envoy\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.720035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.714369 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rcfn\" (UniqueName: \"kubernetes.io/projected/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-kube-api-access-2rcfn\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.743029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.742884 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-token\") pod \"e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:34.896594 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:34.896531 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:35.111034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.110982 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4"] Oct 30 19:38:35.111838 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.111821 5047 cpu_manager.go:414] "containerMap: removing container" podUID="5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" containerName="istio-proxy" Oct 30 19:38:35.111880 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.111840 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b488a6e-9b1c-44c0-8656-0bd32d3e3b96" containerName="istio-proxy" Oct 30 19:38:35.117336 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.117307 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.140459 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.140423 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4"] Oct 30 19:38:35.146952 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.146850 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-19f646\"/\"kube-root-ca.crt\"" Oct 30 19:38:35.146952 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.146858 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-19f646\"/\"openshift-service-ca.crt\"" Oct 30 19:38:35.147493 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.147259 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-19f646\"/\"default-dockercfg-qjwmr\"" Oct 30 19:38:35.150071 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.150038 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-19f646\"/\"test-model-19f646-kserve-self-signed-certs\"" Oct 30 19:38:35.249788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.249735 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h"] Oct 30 19:38:35.260444 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.260406 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-model-cache\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.260622 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.260583 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-tmp-dir\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.260692 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.260669 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl4qs\" (UniqueName: \"kubernetes.io/projected/6dbadedf-fc49-43d8-82af-d7024864cc1f-kube-api-access-gl4qs\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.260807 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.260788 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-dshm\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.260896 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.260877 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dbadedf-fc49-43d8-82af-d7024864cc1f-tls-certs\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.260952 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.260938 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-home\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.266485 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.266439 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:38:35.266609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.266518 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:38:35.266609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.266552 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:38:35.373029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.365209 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl4qs\" (UniqueName: \"kubernetes.io/projected/6dbadedf-fc49-43d8-82af-d7024864cc1f-kube-api-access-gl4qs\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.373029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.365359 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-dshm\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.373029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.365435 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dbadedf-fc49-43d8-82af-d7024864cc1f-tls-certs\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.373029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.365484 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-home\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.373029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.365624 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-model-cache\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.373029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.365766 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-tmp-dir\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.373029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.366407 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-tmp-dir\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.373029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.369401 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-home\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.373029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.371429 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-model-cache\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.391029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.389233 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl4qs\" (UniqueName: \"kubernetes.io/projected/6dbadedf-fc49-43d8-82af-d7024864cc1f-kube-api-access-gl4qs\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.405091 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.404686 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dbadedf-fc49-43d8-82af-d7024864cc1f-tls-certs\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.405091 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.404919 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-dshm\") pod \"test-model-19f646-kserve-6cdc876db7-b5rm4\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.437688 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.433929 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:35.735128 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.728118 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" event={"ID":"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c","Type":"ContainerStarted","Data":"69456a855408608c1513822806b145329f66d796c7f1281804f4aec66378faab"} Oct 30 19:38:35.735128 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.728188 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" event={"ID":"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c","Type":"ContainerStarted","Data":"8f40d8a8af589807b83f7f7a8c02f10c6b8f8db7ae2cc46f9b94f7eb3679b8e2"} Oct 30 19:38:35.832062 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.823755 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" podStartSLOduration=1.823733107 podStartE2EDuration="1.823733107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:35.814461557 +0000 UTC m=+2068.888916915" watchObservedRunningTime="2026-10-30 19:38:35.823733107 +0000 UTC m=+2068.898188465" Oct 30 19:38:35.898027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:35.897148 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:36.868518 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:36.817196 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:38:36.868518 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:36.818375 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:38:36.906634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:36.905899 5047 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:38:36.906634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:36.905972 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" podUID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 19:38:36.916986 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:36.916957 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4"] Oct 30 19:38:36.957604 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:38:36.955904 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dbadedf_fc49_43d8_82af_d7024864cc1f.slice/crio-e3f0077e8dfef05d8a2a21d74adb6f811edc929432a2bcc27847a305a959040c WatchSource:0}: Error finding container e3f0077e8dfef05d8a2a21d74adb6f811edc929432a2bcc27847a305a959040c: Status 404 returned error can't find the container with id e3f0077e8dfef05d8a2a21d74adb6f811edc929432a2bcc27847a305a959040c Oct 30 19:38:37.878402 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:37.878362 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" event={"ID":"6dbadedf-fc49-43d8-82af-d7024864cc1f","Type":"ContainerStarted","Data":"31e1534c761c4fb91e5d1412374d62bb8e3785131381e234439920cf5a6503ac"} Oct 30 19:38:37.878829 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:37.878413 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" event={"ID":"6dbadedf-fc49-43d8-82af-d7024864cc1f","Type":"ContainerStarted","Data":"e3f0077e8dfef05d8a2a21d74adb6f811edc929432a2bcc27847a305a959040c"} Oct 30 19:38:37.902996 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:37.902960 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:37.919855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:37.919803 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" podStartSLOduration=2.9197888 podStartE2EDuration="2.9197888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:37.916413963 +0000 UTC m=+2070.990869320" watchObservedRunningTime="2026-10-30 19:38:37.9197888 +0000 UTC m=+2070.994244192" Oct 30 19:38:38.434634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:38.434594 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:38.886203 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:38.886166 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:38.887357 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:38.887330 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:38:39.602967 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:38:39.602932 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:38:40.615541 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:40.615503 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b"] Oct 30 19:38:40.615981 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:40.615706 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" podUID="581eaa37-1f68-4519-a69a-3f4fcee15057" containerName="istio-proxy" containerID="cri-o://d7186411c8b531586afb6bef871723146d1f7042d10103c8f08c1bcac37e42fe" gracePeriod=30 Oct 30 19:38:41.018529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.018447 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr"] Oct 30 19:38:41.023740 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.023718 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" Oct 30 19:38:41.038457 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.038426 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-08bcfe\"" Oct 30 19:38:41.056061 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.056027 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr"] Oct 30 19:38:41.082736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.082698 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g"] Oct 30 19:38:41.089247 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.089218 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" Oct 30 19:38:41.104206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.104174 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-08bcfe-dockercfg-dlkm9\"" Oct 30 19:38:41.108429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.108400 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g"] Oct 30 19:38:41.229045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.219523 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ljrg\" (UniqueName: \"kubernetes.io/projected/5428f014-f74f-4362-9010-f510a62a5b87-kube-api-access-7ljrg\") pod \"payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g\" (UID: \"5428f014-f74f-4362-9010-f510a62a5b87\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" Oct 30 19:38:41.230612 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.230576 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mrzk\" (UniqueName: \"kubernetes.io/projected/77f27bee-11a8-48c1-bdd9-3f76fafd653e-kube-api-access-2mrzk\") pod \"payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr\" (UID: \"77f27bee-11a8-48c1-bdd9-3f76fafd653e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" Oct 30 19:38:41.230873 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.230853 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5428f014-f74f-4362-9010-f510a62a5b87-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g\" (UID: \"5428f014-f74f-4362-9010-f510a62a5b87\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" Oct 30 19:38:41.247071 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.231049 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f27bee-11a8-48c1-bdd9-3f76fafd653e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr\" (UID: \"77f27bee-11a8-48c1-bdd9-3f76fafd653e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" Oct 30 19:38:41.361608 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.361563 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f27bee-11a8-48c1-bdd9-3f76fafd653e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr\" (UID: \"77f27bee-11a8-48c1-bdd9-3f76fafd653e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" Oct 30 19:38:41.361821 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.361770 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ljrg\" (UniqueName: \"kubernetes.io/projected/5428f014-f74f-4362-9010-f510a62a5b87-kube-api-access-7ljrg\") pod \"payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g\" (UID: \"5428f014-f74f-4362-9010-f510a62a5b87\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" Oct 30 19:38:41.362092 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.362059 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mrzk\" (UniqueName: \"kubernetes.io/projected/77f27bee-11a8-48c1-bdd9-3f76fafd653e-kube-api-access-2mrzk\") pod \"payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr\" (UID: \"77f27bee-11a8-48c1-bdd9-3f76fafd653e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" Oct 30 19:38:41.362190 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.362153 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5428f014-f74f-4362-9010-f510a62a5b87-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g\" (UID: \"5428f014-f74f-4362-9010-f510a62a5b87\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" Oct 30 19:38:41.371032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.362947 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f27bee-11a8-48c1-bdd9-3f76fafd653e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr\" (UID: \"77f27bee-11a8-48c1-bdd9-3f76fafd653e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" Oct 30 19:38:41.371032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.363159 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5428f014-f74f-4362-9010-f510a62a5b87-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g\" (UID: \"5428f014-f74f-4362-9010-f510a62a5b87\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" Oct 30 19:38:41.510670 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.510506 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ljrg\" (UniqueName: \"kubernetes.io/projected/5428f014-f74f-4362-9010-f510a62a5b87-kube-api-access-7ljrg\") pod \"payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g\" (UID: \"5428f014-f74f-4362-9010-f510a62a5b87\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" Oct 30 19:38:41.528201 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.528131 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mrzk\" (UniqueName: \"kubernetes.io/projected/77f27bee-11a8-48c1-bdd9-3f76fafd653e-kube-api-access-2mrzk\") pod \"payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr\" (UID: \"77f27bee-11a8-48c1-bdd9-3f76fafd653e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" Oct 30 19:38:41.638617 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.638540 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" Oct 30 19:38:41.715421 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.712702 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" Oct 30 19:38:41.942182 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:38:41.942155 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f27bee_11a8_48c1_bdd9_3f76fafd653e.slice/crio-42124add7a2c2e257790b677dcfa9d0570d1b2cf21f2c8eb7dc66740eb175c17 WatchSource:0}: Error finding container 42124add7a2c2e257790b677dcfa9d0570d1b2cf21f2c8eb7dc66740eb175c17: Status 404 returned error can't find the container with id 42124add7a2c2e257790b677dcfa9d0570d1b2cf21f2c8eb7dc66740eb175c17 Oct 30 19:38:41.943899 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:41.943871 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr"] Oct 30 19:38:42.050335 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:38:42.050307 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5428f014_f74f_4362_9010_f510a62a5b87.slice/crio-7457e1d0e11c3cca165f4e2a429f0cd5b1947d0c77cb1a60d05918aad5f3dd8b WatchSource:0}: Error finding container 7457e1d0e11c3cca165f4e2a429f0cd5b1947d0c77cb1a60d05918aad5f3dd8b: Status 404 returned error can't find the container with id 7457e1d0e11c3cca165f4e2a429f0cd5b1947d0c77cb1a60d05918aad5f3dd8b Oct 30 19:38:42.052252 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:42.052226 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g"] Oct 30 19:38:42.922459 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:42.922420 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" event={"ID":"5428f014-f74f-4362-9010-f510a62a5b87","Type":"ContainerStarted","Data":"003ed777e3cbb386c4a1a030051e0ffe4e06212c611737ee6cd128349b85bb4b"} Oct 30 19:38:42.922946 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:42.922509 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" event={"ID":"5428f014-f74f-4362-9010-f510a62a5b87","Type":"ContainerStarted","Data":"7457e1d0e11c3cca165f4e2a429f0cd5b1947d0c77cb1a60d05918aad5f3dd8b"} Oct 30 19:38:42.922946 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:42.922678 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" Oct 30 19:38:42.924361 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:42.924337 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" event={"ID":"77f27bee-11a8-48c1-bdd9-3f76fafd653e","Type":"ContainerStarted","Data":"902e5053fa6f6537851ca9e0e000dd7e2e7268f7c02b0d7908838414cc8288e0"} Oct 30 19:38:42.924484 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:42.924380 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" Oct 30 19:38:42.924484 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:42.924397 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" event={"ID":"77f27bee-11a8-48c1-bdd9-3f76fafd653e","Type":"ContainerStarted","Data":"42124add7a2c2e257790b677dcfa9d0570d1b2cf21f2c8eb7dc66740eb175c17"} Oct 30 19:38:42.959551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:42.959499 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" podStartSLOduration=1.959484279 podStartE2EDuration="1.959484279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:42.958518907 +0000 UTC m=+2076.032974264" watchObservedRunningTime="2026-10-30 19:38:42.959484279 +0000 UTC m=+2076.033939647" Oct 30 19:38:42.999609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:42.999559 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" podStartSLOduration=2.999545182 podStartE2EDuration="2.999545182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:42.99726286 +0000 UTC m=+2076.071718218" watchObservedRunningTime="2026-10-30 19:38:42.999545182 +0000 UTC m=+2076.074000537" Oct 30 19:38:45.435098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:45.435066 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:45.447839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:45.447808 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:46.152028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.150830 5047 generic.go:356] "Generic (PLEG): container finished" podID="581eaa37-1f68-4519-a69a-3f4fcee15057" containerID="d7186411c8b531586afb6bef871723146d1f7042d10103c8f08c1bcac37e42fe" exitCode=0 Oct 30 19:38:46.171155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.161685 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" event={"ID":"581eaa37-1f68-4519-a69a-3f4fcee15057","Type":"ContainerDied","Data":"d7186411c8b531586afb6bef871723146d1f7042d10103c8f08c1bcac37e42fe"} Oct 30 19:38:46.171155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.161775 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" event={"ID":"581eaa37-1f68-4519-a69a-3f4fcee15057","Type":"ContainerDied","Data":"78045f39d87766faf2c787da5d1616d77c6d20fb11cc663ed3168a04b6d09ec8"} Oct 30 19:38:46.171155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.161800 5047 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78045f39d87766faf2c787da5d1616d77c6d20fb11cc663ed3168a04b6d09ec8" Oct 30 19:38:46.193661 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.193632 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:38:46.276098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.270818 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-token\" (UniqueName: \"kubernetes.io/projected/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-token\") pod \"581eaa37-1f68-4519-a69a-3f4fcee15057\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " Oct 30 19:38:46.276098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.270970 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-data\") pod \"581eaa37-1f68-4519-a69a-3f4fcee15057\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " Oct 30 19:38:46.276098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.271025 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-credential-socket\") pod \"581eaa37-1f68-4519-a69a-3f4fcee15057\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " Oct 30 19:38:46.276098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.271109 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-envoy\") pod \"581eaa37-1f68-4519-a69a-3f4fcee15057\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " Oct 30 19:38:46.276098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.271141 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/581eaa37-1f68-4519-a69a-3f4fcee15057-kube-api-access-nfrb4\" (UniqueName: \"kubernetes.io/projected/581eaa37-1f68-4519-a69a-3f4fcee15057-kube-api-access-nfrb4\") pod \"581eaa37-1f68-4519-a69a-3f4fcee15057\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " Oct 30 19:38:46.276098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.271406 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-workload-certs\") pod \"581eaa37-1f68-4519-a69a-3f4fcee15057\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " Oct 30 19:38:46.276098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.271498 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/581eaa37-1f68-4519-a69a-3f4fcee15057-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/581eaa37-1f68-4519-a69a-3f4fcee15057-istiod-ca-cert\") pod \"581eaa37-1f68-4519-a69a-3f4fcee15057\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " Oct 30 19:38:46.276098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.272481 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-workload-socket" pod "581eaa37-1f68-4519-a69a-3f4fcee15057" (UID: "581eaa37-1f68-4519-a69a-3f4fcee15057"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:46.276098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.273124 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-workload-socket\") pod \"581eaa37-1f68-4519-a69a-3f4fcee15057\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " Oct 30 19:38:46.276098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.273191 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-podinfo\") pod \"581eaa37-1f68-4519-a69a-3f4fcee15057\" (UID: \"581eaa37-1f68-4519-a69a-3f4fcee15057\") " Oct 30 19:38:46.276098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.274253 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:46.351716 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.350738 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-credential-socket" pod "581eaa37-1f68-4519-a69a-3f4fcee15057" (UID: "581eaa37-1f68-4519-a69a-3f4fcee15057"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:46.351716 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.351386 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-workload-certs" pod "581eaa37-1f68-4519-a69a-3f4fcee15057" (UID: "581eaa37-1f68-4519-a69a-3f4fcee15057"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:46.373074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.354849 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/581eaa37-1f68-4519-a69a-3f4fcee15057-kube-api-access-nfrb4" pod "581eaa37-1f68-4519-a69a-3f4fcee15057" (UID: "581eaa37-1f68-4519-a69a-3f4fcee15057"). InnerVolumeSpecName "kube-api-access-nfrb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:46.393032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.385221 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:46.393032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.385264 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:46.393032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.385280 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfrb4\" (UniqueName: \"kubernetes.io/projected/581eaa37-1f68-4519-a69a-3f4fcee15057-kube-api-access-nfrb4\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:46.407422 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.405279 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/581eaa37-1f68-4519-a69a-3f4fcee15057-istiod-ca-cert" pod "581eaa37-1f68-4519-a69a-3f4fcee15057" (UID: "581eaa37-1f68-4519-a69a-3f4fcee15057"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:46.421303 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.420970 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:38:46.422993 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.422960 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-data" pod "581eaa37-1f68-4519-a69a-3f4fcee15057" (UID: "581eaa37-1f68-4519-a69a-3f4fcee15057"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:46.429911 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.429878 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-podinfo" pod "581eaa37-1f68-4519-a69a-3f4fcee15057" (UID: "581eaa37-1f68-4519-a69a-3f4fcee15057"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:38:46.456159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.456120 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-envoy" pod "581eaa37-1f68-4519-a69a-3f4fcee15057" (UID: "581eaa37-1f68-4519-a69a-3f4fcee15057"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:46.473147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.468995 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-token" pod "581eaa37-1f68-4519-a69a-3f4fcee15057" (UID: "581eaa37-1f68-4519-a69a-3f4fcee15057"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:46.493994 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.493961 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/581eaa37-1f68-4519-a69a-3f4fcee15057-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:46.494318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.494303 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:46.494447 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.494435 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:46.494621 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.494611 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:46.494718 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:46.494709 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/581eaa37-1f68-4519-a69a-3f4fcee15057-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:38:47.171763 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:47.171722 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b" Oct 30 19:38:47.264318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:47.264260 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b"] Oct 30 19:38:47.268945 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:47.268908 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-xgc8b"] Oct 30 19:38:49.071851 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:49.071817 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="581eaa37-1f68-4519-a69a-3f4fcee15057" path="/var/lib/kubelet/pods/581eaa37-1f68-4519-a69a-3f4fcee15057/volumes" Oct 30 19:38:53.935778 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:53.935742 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" Oct 30 19:38:53.936233 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:53.935833 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" Oct 30 19:38:55.307211 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:55.307151 5047 patch_prober.go:29] interesting pod/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:38:55.307632 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:55.307235 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" podUID="9859e88e-b269-42ac-8647-0bf70d7080cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:38:57.291145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:57.291107 5047 generic.go:356] "Generic (PLEG): container finished" podID="9859e88e-b269-42ac-8647-0bf70d7080cb" containerID="f3126db0e1961ffb3fa4ed39c1beecef6c7018289741a6b340968eebcfb38bb2" exitCode=0 Oct 30 19:38:57.291603 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:57.291174 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" event={"ID":"9859e88e-b269-42ac-8647-0bf70d7080cb","Type":"ContainerDied","Data":"f3126db0e1961ffb3fa4ed39c1beecef6c7018289741a6b340968eebcfb38bb2"} Oct 30 19:38:57.291684 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:57.291659 5047 scope.go:122] "RemoveContainer" containerID="f3126db0e1961ffb3fa4ed39c1beecef6c7018289741a6b340968eebcfb38bb2" Oct 30 19:38:57.293485 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:57.293452 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:38:57.293578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:57.293528 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:38:57.293578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:57.293564 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:38:58.311433 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:58.311393 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" event={"ID":"9859e88e-b269-42ac-8647-0bf70d7080cb","Type":"ContainerStarted","Data":"7b668b4cf4b747eb55dcf1e6bcf0e21419004f22cb478f94b7af161b10ee6eba"} Oct 30 19:38:59.159276 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:59.159191 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:38:59.160320 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:59.160267 5047 patch_prober.go:29] interesting pod/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:38:59.160608 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:59.160576 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" podUID="9859e88e-b269-42ac-8647-0bf70d7080cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 19:38:59.327694 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:38:59.327652 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25"] Oct 30 19:39:00.143071 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:00.143031 5047 patch_prober.go:29] interesting pod/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:39:00.143379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:00.143344 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" podUID="9859e88e-b269-42ac-8647-0bf70d7080cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 19:39:01.159516 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:01.159475 5047 patch_prober.go:29] interesting pod/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:39:01.160190 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:01.160156 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" podUID="9859e88e-b269-42ac-8647-0bf70d7080cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 19:39:02.155531 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:02.155486 5047 patch_prober.go:29] interesting pod/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:39:02.155855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:02.155824 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" podUID="9859e88e-b269-42ac-8647-0bf70d7080cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 19:39:03.159280 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:03.159237 5047 patch_prober.go:29] interesting pod/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:39:03.159857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:03.159825 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" podUID="9859e88e-b269-42ac-8647-0bf70d7080cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 19:39:03.304116 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:03.303863 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq"] Oct 30 19:39:03.304312 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:03.304128 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" podUID="c10e3c6c-bbf1-4606-990e-bc2898aff682" containerName="istio-proxy" containerID="cri-o://446d0473ff1c0841ad2cf8676751bc7fa653ec3f6f22515dffffb4620ae987f5" gracePeriod=30 Oct 30 19:39:04.222030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.216857 5047 patch_prober.go:29] interesting pod/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:39:04.222030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.216919 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" podUID="9859e88e-b269-42ac-8647-0bf70d7080cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 19:39:04.328436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.328383 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" podUID="d86a88d1-aeee-4dd6-bbc3-a627fdd4854c" containerName="payload-pre-processing" containerID="cri-o://3826a3cbb86d2ce0a39335345f6607e2237ce6469e594dcdd1cf3621f55a1cc0" gracePeriod=25 Oct 30 19:39:04.610637 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.606225 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh"] Oct 30 19:39:04.610637 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.607546 5047 cpu_manager.go:414] "containerMap: removing container" podUID="581eaa37-1f68-4519-a69a-3f4fcee15057" containerName="istio-proxy" Oct 30 19:39:04.610637 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.607568 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="581eaa37-1f68-4519-a69a-3f4fcee15057" containerName="istio-proxy" Oct 30 19:39:04.610637 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.607765 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="581eaa37-1f68-4519-a69a-3f4fcee15057" containerName="istio-proxy" Oct 30 19:39:04.611365 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.611339 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.636226 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.636189 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh"] Oct 30 19:39:04.639228 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.639202 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-614c02-openshift-default-dockercfg-hjlgf\"" Oct 30 19:39:04.661734 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.661701 5047 generic.go:356] "Generic (PLEG): container finished" podID="d86a88d1-aeee-4dd6-bbc3-a627fdd4854c" containerID="3826a3cbb86d2ce0a39335345f6607e2237ce6469e594dcdd1cf3621f55a1cc0" exitCode=0 Oct 30 19:39:04.661911 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.661755 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" event={"ID":"d86a88d1-aeee-4dd6-bbc3-a627fdd4854c","Type":"ContainerDied","Data":"3826a3cbb86d2ce0a39335345f6607e2237ce6469e594dcdd1cf3621f55a1cc0"} Oct 30 19:39:04.661911 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.661793 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" event={"ID":"d86a88d1-aeee-4dd6-bbc3-a627fdd4854c","Type":"ContainerDied","Data":"ab453cd8a0fd1d54dffc74bbbd0d4ee1d696a1fbbc67384eeaaa19fb57d9dae5"} Oct 30 19:39:04.661911 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.661814 5047 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab453cd8a0fd1d54dffc74bbbd0d4ee1d696a1fbbc67384eeaaa19fb57d9dae5" Oct 30 19:39:04.671440 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.671402 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istiod-ca-cert\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.671643 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.671626 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-envoy\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.671763 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.671750 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-credential-socket\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.671877 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.671864 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-data\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.671995 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.671982 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cfst\" (UniqueName: \"kubernetes.io/projected/96bf353d-9dac-4c62-ac9e-c97d305d08a4-kube-api-access-5cfst\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.672218 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.672203 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-podinfo\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.672328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.672316 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-token\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.672609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.672590 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-workload-socket\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.672795 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.672777 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-workload-certs\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.680481 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.680455 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" Oct 30 19:39:04.792883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.790427 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d86a88d1-aeee-4dd6-bbc3-a627fdd4854c-kube-api-access-kh9gp\" (UniqueName: \"kubernetes.io/projected/d86a88d1-aeee-4dd6-bbc3-a627fdd4854c-kube-api-access-kh9gp\") pod \"d86a88d1-aeee-4dd6-bbc3-a627fdd4854c\" (UID: \"d86a88d1-aeee-4dd6-bbc3-a627fdd4854c\") " Oct 30 19:39:04.792883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.790513 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d86a88d1-aeee-4dd6-bbc3-a627fdd4854c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86a88d1-aeee-4dd6-bbc3-a627fdd4854c-plugins-config-volume\") pod \"d86a88d1-aeee-4dd6-bbc3-a627fdd4854c\" (UID: \"d86a88d1-aeee-4dd6-bbc3-a627fdd4854c\") " Oct 30 19:39:04.792883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.791129 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-podinfo\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.792883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.791186 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-token\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.792883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.791425 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-workload-socket\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.792883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.791520 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-workload-certs\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.792883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.791593 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istiod-ca-cert\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.792883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.791622 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-envoy\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.792883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.791653 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-credential-socket\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.792883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.791694 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-data\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.792883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.791746 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cfst\" (UniqueName: \"kubernetes.io/projected/96bf353d-9dac-4c62-ac9e-c97d305d08a4-kube-api-access-5cfst\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.792883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.792489 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-workload-certs\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.804115 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.802251 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86a88d1-aeee-4dd6-bbc3-a627fdd4854c-plugins-config-volume" pod "d86a88d1-aeee-4dd6-bbc3-a627fdd4854c" (UID: "d86a88d1-aeee-4dd6-bbc3-a627fdd4854c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:04.804115 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.802628 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-workload-socket\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.804115 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.803373 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-credential-socket\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.804115 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.803633 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-data\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.814044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.812346 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istiod-ca-cert\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.849054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.844482 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-envoy\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.849054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.845479 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-token\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:04.919445 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:04.919410 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86a88d1-aeee-4dd6-bbc3-a627fdd4854c-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:39:05.051107 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:05.050869 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-podinfo\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:05.052057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:05.051423 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86a88d1-aeee-4dd6-bbc3-a627fdd4854c-kube-api-access-kh9gp" pod "d86a88d1-aeee-4dd6-bbc3-a627fdd4854c" (UID: "d86a88d1-aeee-4dd6-bbc3-a627fdd4854c"). InnerVolumeSpecName "kube-api-access-kh9gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:05.057958 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:05.057897 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cfst\" (UniqueName: \"kubernetes.io/projected/96bf353d-9dac-4c62-ac9e-c97d305d08a4-kube-api-access-5cfst\") pod \"e2e-inf-a-614c02-openshift-default-5984c4975-frdxh\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:05.159917 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:05.159864 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:39:05.162448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:05.161740 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:39:05.163634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:05.163602 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh9gp\" (UniqueName: \"kubernetes.io/projected/d86a88d1-aeee-4dd6-bbc3-a627fdd4854c-kube-api-access-kh9gp\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:39:05.169166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:05.169145 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:39:05.279474 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:05.279437 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:05.607030 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:39:05.606987 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96bf353d_9dac_4c62_ac9e_c97d305d08a4.slice/crio-7613c7f9054d8a61dfc56bb6087390c51445a92cace340e964aa6a8439294995 WatchSource:0}: Error finding container 7613c7f9054d8a61dfc56bb6087390c51445a92cace340e964aa6a8439294995: Status 404 returned error can't find the container with id 7613c7f9054d8a61dfc56bb6087390c51445a92cace340e964aa6a8439294995 Oct 30 19:39:05.609672 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:05.609644 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh"] Oct 30 19:39:05.610052 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:05.609985 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:39:05.610108 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:05.610081 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:39:05.610156 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:05.610115 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:39:05.674911 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:05.674875 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" event={"ID":"96bf353d-9dac-4c62-ac9e-c97d305d08a4","Type":"ContainerStarted","Data":"7613c7f9054d8a61dfc56bb6087390c51445a92cace340e964aa6a8439294995"} Oct 30 19:39:05.675176 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:05.675075 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25" Oct 30 19:39:05.727603 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:05.727571 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25"] Oct 30 19:39:05.734559 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:05.734526 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4cb13-75cf5fbfdc-j5z25"] Oct 30 19:39:06.685300 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:06.685259 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" event={"ID":"96bf353d-9dac-4c62-ac9e-c97d305d08a4","Type":"ContainerStarted","Data":"f484110505d1ebb6d86ee82833d9148eb10a96b87236e75a4f02ce47931e44de"} Oct 30 19:39:06.724842 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:06.724787 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" podStartSLOduration=2.724772918 podStartE2EDuration="2.724772918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:06.723842084 +0000 UTC m=+2099.798297466" watchObservedRunningTime="2026-10-30 19:39:06.724772918 +0000 UTC m=+2099.799228276" Oct 30 19:39:06.775417 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:06.775370 5047 patch_prober.go:29] interesting pod/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:39:06.775626 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:06.775449 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" podUID="c10e3c6c-bbf1-4606-990e-bc2898aff682" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:39:07.129149 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:07.129110 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d86a88d1-aeee-4dd6-bbc3-a627fdd4854c" path="/var/lib/kubelet/pods/d86a88d1-aeee-4dd6-bbc3-a627fdd4854c/volumes" Oct 30 19:39:07.280487 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:07.280451 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:07.286862 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:07.286834 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:07.644062 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:07.644030 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:39:07.650775 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:07.650748 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:39:07.694712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:07.694682 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:07.695845 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:07.695823 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:39:08.629207 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.629179 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:39:08.678676 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.678640 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-podinfo\") pod \"c10e3c6c-bbf1-4606-990e-bc2898aff682\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " Oct 30 19:39:08.678993 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.678696 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c10e3c6c-bbf1-4606-990e-bc2898aff682-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10e3c6c-bbf1-4606-990e-bc2898aff682-istiod-ca-cert\") pod \"c10e3c6c-bbf1-4606-990e-bc2898aff682\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " Oct 30 19:39:08.678993 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.678932 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10e3c6c-bbf1-4606-990e-bc2898aff682-kube-api-access-vj2pd\" (UniqueName: \"kubernetes.io/projected/c10e3c6c-bbf1-4606-990e-bc2898aff682-kube-api-access-vj2pd\") pod \"c10e3c6c-bbf1-4606-990e-bc2898aff682\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " Oct 30 19:39:08.679370 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.679017 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-token\" (UniqueName: \"kubernetes.io/projected/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-token\") pod \"c10e3c6c-bbf1-4606-990e-bc2898aff682\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " Oct 30 19:39:08.679370 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.679112 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-data\") pod \"c10e3c6c-bbf1-4606-990e-bc2898aff682\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " Oct 30 19:39:08.679370 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.679241 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-envoy\") pod \"c10e3c6c-bbf1-4606-990e-bc2898aff682\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " Oct 30 19:39:08.679592 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.679397 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-workload-certs\") pod \"c10e3c6c-bbf1-4606-990e-bc2898aff682\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " Oct 30 19:39:08.679592 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.679441 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10e3c6c-bbf1-4606-990e-bc2898aff682-istiod-ca-cert" pod "c10e3c6c-bbf1-4606-990e-bc2898aff682" (UID: "c10e3c6c-bbf1-4606-990e-bc2898aff682"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:08.679592 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.679452 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-credential-socket\") pod \"c10e3c6c-bbf1-4606-990e-bc2898aff682\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " Oct 30 19:39:08.679756 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.679595 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-workload-socket\") pod \"c10e3c6c-bbf1-4606-990e-bc2898aff682\" (UID: \"c10e3c6c-bbf1-4606-990e-bc2898aff682\") " Oct 30 19:39:08.679756 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.679736 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-credential-socket" pod "c10e3c6c-bbf1-4606-990e-bc2898aff682" (UID: "c10e3c6c-bbf1-4606-990e-bc2898aff682"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:08.680925 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.680797 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-data" pod "c10e3c6c-bbf1-4606-990e-bc2898aff682" (UID: "c10e3c6c-bbf1-4606-990e-bc2898aff682"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:08.682563 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.682523 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-workload-certs" pod "c10e3c6c-bbf1-4606-990e-bc2898aff682" (UID: "c10e3c6c-bbf1-4606-990e-bc2898aff682"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:08.682835 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.682771 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-token" pod "c10e3c6c-bbf1-4606-990e-bc2898aff682" (UID: "c10e3c6c-bbf1-4606-990e-bc2898aff682"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:08.682974 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.682947 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-workload-socket" pod "c10e3c6c-bbf1-4606-990e-bc2898aff682" (UID: "c10e3c6c-bbf1-4606-990e-bc2898aff682"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:08.695568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.695533 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.695568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.695571 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.696328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.695588 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.696328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.695602 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.696328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.695616 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.696328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.695631 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10e3c6c-bbf1-4606-990e-bc2898aff682-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.697682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.697621 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-podinfo" pod "c10e3c6c-bbf1-4606-990e-bc2898aff682" (UID: "c10e3c6c-bbf1-4606-990e-bc2898aff682"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:39:08.702130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.699682 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10e3c6c-bbf1-4606-990e-bc2898aff682-kube-api-access-vj2pd" pod "c10e3c6c-bbf1-4606-990e-bc2898aff682" (UID: "c10e3c6c-bbf1-4606-990e-bc2898aff682"). InnerVolumeSpecName "kube-api-access-vj2pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:08.702130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.700434 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-envoy" pod "c10e3c6c-bbf1-4606-990e-bc2898aff682" (UID: "c10e3c6c-bbf1-4606-990e-bc2898aff682"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:08.714569 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.714534 5047 generic.go:356] "Generic (PLEG): container finished" podID="c10e3c6c-bbf1-4606-990e-bc2898aff682" containerID="446d0473ff1c0841ad2cf8676751bc7fa653ec3f6f22515dffffb4620ae987f5" exitCode=0 Oct 30 19:39:08.715630 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.715606 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" Oct 30 19:39:08.718824 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.718787 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" event={"ID":"c10e3c6c-bbf1-4606-990e-bc2898aff682","Type":"ContainerDied","Data":"446d0473ff1c0841ad2cf8676751bc7fa653ec3f6f22515dffffb4620ae987f5"} Oct 30 19:39:08.718985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.718851 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq" event={"ID":"c10e3c6c-bbf1-4606-990e-bc2898aff682","Type":"ContainerDied","Data":"6d4ebc3631b8915edde2d1b34fd3c0a264076521ba7be2b5051d162398cfc732"} Oct 30 19:39:08.718985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.718880 5047 scope.go:122] "RemoveContainer" containerID="446d0473ff1c0841ad2cf8676751bc7fa653ec3f6f22515dffffb4620ae987f5" Oct 30 19:39:08.744189 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.744163 5047 scope.go:122] "RemoveContainer" containerID="29a21cb49fc2a3a1f5364edbc0955a65f06c05ab088daede9233aefc759f059f" Oct 30 19:39:08.786800 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.786767 5047 scope.go:122] "RemoveContainer" containerID="446d0473ff1c0841ad2cf8676751bc7fa653ec3f6f22515dffffb4620ae987f5" Oct 30 19:39:08.787497 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:39:08.787463 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"446d0473ff1c0841ad2cf8676751bc7fa653ec3f6f22515dffffb4620ae987f5\": container with ID starting with 446d0473ff1c0841ad2cf8676751bc7fa653ec3f6f22515dffffb4620ae987f5 not found: ID does not exist" containerID="446d0473ff1c0841ad2cf8676751bc7fa653ec3f6f22515dffffb4620ae987f5" Oct 30 19:39:08.787650 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.787508 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"446d0473ff1c0841ad2cf8676751bc7fa653ec3f6f22515dffffb4620ae987f5"} err="failed to get container status \"446d0473ff1c0841ad2cf8676751bc7fa653ec3f6f22515dffffb4620ae987f5\": rpc error: code = NotFound desc = could not find container \"446d0473ff1c0841ad2cf8676751bc7fa653ec3f6f22515dffffb4620ae987f5\": container with ID starting with 446d0473ff1c0841ad2cf8676751bc7fa653ec3f6f22515dffffb4620ae987f5 not found: ID does not exist" Oct 30 19:39:08.787650 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.787536 5047 scope.go:122] "RemoveContainer" containerID="29a21cb49fc2a3a1f5364edbc0955a65f06c05ab088daede9233aefc759f059f" Oct 30 19:39:08.787801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.787763 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq"] Oct 30 19:39:08.787879 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:39:08.787857 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a21cb49fc2a3a1f5364edbc0955a65f06c05ab088daede9233aefc759f059f\": container with ID starting with 29a21cb49fc2a3a1f5364edbc0955a65f06c05ab088daede9233aefc759f059f not found: ID does not exist" containerID="29a21cb49fc2a3a1f5364edbc0955a65f06c05ab088daede9233aefc759f059f" Oct 30 19:39:08.787929 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.787888 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a21cb49fc2a3a1f5364edbc0955a65f06c05ab088daede9233aefc759f059f"} err="failed to get container status \"29a21cb49fc2a3a1f5364edbc0955a65f06c05ab088daede9233aefc759f059f\": rpc error: code = NotFound desc = could not find container \"29a21cb49fc2a3a1f5364edbc0955a65f06c05ab088daede9233aefc759f059f\": container with ID starting with 29a21cb49fc2a3a1f5364edbc0955a65f06c05ab088daede9233aefc759f059f not found: ID does not exist" Oct 30 19:39:08.792932 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.792907 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b625eb-openshift-default-7df9dd974c-k9wzq"] Oct 30 19:39:08.797934 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.797910 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.797934 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.797930 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10e3c6c-bbf1-4606-990e-bc2898aff682-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:39:08.798142 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:08.797940 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj2pd\" (UniqueName: \"kubernetes.io/projected/c10e3c6c-bbf1-4606-990e-bc2898aff682-kube-api-access-vj2pd\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:39:09.055277 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:09.055201 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c10e3c6c-bbf1-4606-990e-bc2898aff682" path="/var/lib/kubelet/pods/c10e3c6c-bbf1-4606-990e-bc2898aff682/volumes" Oct 30 19:39:09.603622 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:39:09.603584 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:39:19.083417 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.083327 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z"] Oct 30 19:39:19.084249 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.084229 5047 cpu_manager.go:414] "containerMap: removing container" podUID="d86a88d1-aeee-4dd6-bbc3-a627fdd4854c" containerName="payload-pre-processing" Oct 30 19:39:19.084301 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.084251 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d86a88d1-aeee-4dd6-bbc3-a627fdd4854c" containerName="payload-pre-processing" Oct 30 19:39:19.084301 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.084272 5047 cpu_manager.go:414] "containerMap: removing container" podUID="c10e3c6c-bbf1-4606-990e-bc2898aff682" containerName="istio-proxy" Oct 30 19:39:19.084301 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.084278 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10e3c6c-bbf1-4606-990e-bc2898aff682" containerName="istio-proxy" Oct 30 19:39:19.084301 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.084288 5047 cpu_manager.go:414] "containerMap: removing container" podUID="c10e3c6c-bbf1-4606-990e-bc2898aff682" containerName="istio-proxy" Oct 30 19:39:19.084301 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.084292 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10e3c6c-bbf1-4606-990e-bc2898aff682" containerName="istio-proxy" Oct 30 19:39:19.084462 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.084394 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10e3c6c-bbf1-4606-990e-bc2898aff682" containerName="istio-proxy" Oct 30 19:39:19.084462 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.084402 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10e3c6c-bbf1-4606-990e-bc2898aff682" containerName="istio-proxy" Oct 30 19:39:19.084462 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.084418 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="d86a88d1-aeee-4dd6-bbc3-a627fdd4854c" containerName="payload-pre-processing" Oct 30 19:39:19.090655 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.090631 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:39:19.109248 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.109217 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z"] Oct 30 19:39:19.114867 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.114833 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-08bcfe-serving-cert\"" Oct 30 19:39:19.204582 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.204539 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/484526c0-1f9b-4b29-b26a-192ba5837d58-maas-api-tls\") pod \"maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z\" (UID: \"484526c0-1f9b-4b29-b26a-192ba5837d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:39:19.204773 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.204753 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/484526c0-1f9b-4b29-b26a-192ba5837d58-metrics-certs\") pod \"maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z\" (UID: \"484526c0-1f9b-4b29-b26a-192ba5837d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:39:19.204889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.204871 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r749\" (UniqueName: \"kubernetes.io/projected/484526c0-1f9b-4b29-b26a-192ba5837d58-kube-api-access-8r749\") pod \"maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z\" (UID: \"484526c0-1f9b-4b29-b26a-192ba5837d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:39:19.205049 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.204997 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/484526c0-1f9b-4b29-b26a-192ba5837d58-openshift-service-ca\") pod \"maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z\" (UID: \"484526c0-1f9b-4b29-b26a-192ba5837d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:39:19.310364 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.310317 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/484526c0-1f9b-4b29-b26a-192ba5837d58-maas-api-tls\") pod \"maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z\" (UID: \"484526c0-1f9b-4b29-b26a-192ba5837d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:39:19.340800 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.317211 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/484526c0-1f9b-4b29-b26a-192ba5837d58-maas-api-tls\") pod \"maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z\" (UID: \"484526c0-1f9b-4b29-b26a-192ba5837d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:39:19.340800 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.338407 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/484526c0-1f9b-4b29-b26a-192ba5837d58-metrics-certs\") pod \"maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z\" (UID: \"484526c0-1f9b-4b29-b26a-192ba5837d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:39:19.340800 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.340118 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r749\" (UniqueName: \"kubernetes.io/projected/484526c0-1f9b-4b29-b26a-192ba5837d58-kube-api-access-8r749\") pod \"maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z\" (UID: \"484526c0-1f9b-4b29-b26a-192ba5837d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:39:19.346948 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.346859 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/484526c0-1f9b-4b29-b26a-192ba5837d58-openshift-service-ca\") pod \"maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z\" (UID: \"484526c0-1f9b-4b29-b26a-192ba5837d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:39:19.347839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.347806 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/484526c0-1f9b-4b29-b26a-192ba5837d58-metrics-certs\") pod \"maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z\" (UID: \"484526c0-1f9b-4b29-b26a-192ba5837d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:39:19.354895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.354857 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/484526c0-1f9b-4b29-b26a-192ba5837d58-openshift-service-ca\") pod \"maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z\" (UID: \"484526c0-1f9b-4b29-b26a-192ba5837d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:39:19.376047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.369786 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r749\" (UniqueName: \"kubernetes.io/projected/484526c0-1f9b-4b29-b26a-192ba5837d58-kube-api-access-8r749\") pod \"maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z\" (UID: \"484526c0-1f9b-4b29-b26a-192ba5837d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:39:19.408458 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.408316 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:39:19.739574 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:39:19.739552 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484526c0_1f9b_4b29_b26a_192ba5837d58.slice/crio-a3d6f970b0566df096104ef35cca7885e798f4652df9f654d0354544560607da WatchSource:0}: Error finding container a3d6f970b0566df096104ef35cca7885e798f4652df9f654d0354544560607da: Status 404 returned error can't find the container with id a3d6f970b0566df096104ef35cca7885e798f4652df9f654d0354544560607da Oct 30 19:39:19.740299 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.740269 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z"] Oct 30 19:39:19.740532 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.740512 5047 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:39:19.812467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:19.812431 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" event={"ID":"484526c0-1f9b-4b29-b26a-192ba5837d58","Type":"ContainerStarted","Data":"a3d6f970b0566df096104ef35cca7885e798f4652df9f654d0354544560607da"} Oct 30 19:39:20.823123 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:20.823090 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" event={"ID":"484526c0-1f9b-4b29-b26a-192ba5837d58","Type":"ContainerStarted","Data":"1e65157962e12e762e772412044b89cc4bc04fc0c0a40755c68570e43772e2f2"} Oct 30 19:39:20.823629 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:20.823217 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:39:20.864021 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:20.863950 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" podStartSLOduration=1.510706984 podStartE2EDuration="1.86393659s" totalImagesPullingTime="353.229606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:20.860035793 +0000 UTC m=+2113.934491147" watchObservedRunningTime="2026-10-30 19:39:20.86393659 +0000 UTC m=+2113.938391947" Oct 30 19:39:26.928046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:26.927301 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:39:28.212031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:28.209566 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" podUID="7456b22a-9c80-4fbb-9236-08eed4067902" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": context deadline exceeded" Oct 30 19:39:28.808222 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:28.808164 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:39:29.379634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:29.362736 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:39:29.829083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:29.828122 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:39:29.829283 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:29.829236 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:39:29.892237 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:29.889355 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" podUID="68ea348d-f2bf-4ee5-9062-9649b71be4c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.140:8000/ready\": context deadline exceeded" Oct 30 19:39:30.092528 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:30.092341 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" podUID="7456b22a-9c80-4fbb-9236-08eed4067902" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:39:30.178618 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:30.175525 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" podUID="6dbadedf-fc49-43d8-82af-d7024864cc1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.146:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:39:30.344044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:30.343921 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 30 19:39:37.165153 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:37.165075 5047 generic.go:356] "Generic (PLEG): container finished" podID="c4296edd-bfda-4c0d-898a-55bb6c711682" containerID="6463058d753528fe240ad999bc35d4bad4f45e6f06fbbf55e9206ad0c8ad1956" exitCode=0 Oct 30 19:39:37.165566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:37.165151 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" event={"ID":"c4296edd-bfda-4c0d-898a-55bb6c711682","Type":"ContainerDied","Data":"6463058d753528fe240ad999bc35d4bad4f45e6f06fbbf55e9206ad0c8ad1956"} Oct 30 19:39:37.165675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:37.165658 5047 scope.go:122] "RemoveContainer" containerID="6463058d753528fe240ad999bc35d4bad4f45e6f06fbbf55e9206ad0c8ad1956" Oct 30 19:39:37.167674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:37.167645 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:39:37.167753 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:37.167717 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:39:37.167789 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:37.167753 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:39:38.175888 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:38.175848 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" event={"ID":"c4296edd-bfda-4c0d-898a-55bb6c711682","Type":"ContainerStarted","Data":"eef2ff9876305455713248e00e4c060cae7867cef7ae47f810b6b06d76f64c38"} Oct 30 19:39:38.267253 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:38.267217 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:39:38.268245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:38.268220 5047 patch_prober.go:29] interesting pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:39:38.268368 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:38.268265 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" podUID="c4296edd-bfda-4c0d-898a-55bb6c711682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 19:39:39.252027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:39.238116 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:39:39.267353 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:39.267316 5047 patch_prober.go:29] interesting pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:39:39.267526 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:39.267373 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" podUID="c4296edd-bfda-4c0d-898a-55bb6c711682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 19:39:39.605050 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:39:39.605000 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:39:40.267884 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:40.267845 5047 patch_prober.go:29] interesting pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:39:40.268366 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:40.267901 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" podUID="c4296edd-bfda-4c0d-898a-55bb6c711682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 19:39:41.267278 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:41.267232 5047 patch_prober.go:29] interesting pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:39:41.267460 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:41.267298 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" podUID="c4296edd-bfda-4c0d-898a-55bb6c711682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 19:39:42.267557 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:42.267520 5047 patch_prober.go:29] interesting pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:39:42.267969 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:42.267587 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" podUID="c4296edd-bfda-4c0d-898a-55bb6c711682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 19:39:43.267955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:43.267920 5047 patch_prober.go:29] interesting pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:39:43.268392 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:43.267990 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" podUID="c4296edd-bfda-4c0d-898a-55bb6c711682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 19:39:44.272985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:44.272946 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:39:44.274328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:39:44.274301 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:40:05.340691 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.340651 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9"] Oct 30 19:40:05.341243 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.340894 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" podUID="eca5e1e9-76eb-4731-8041-1f521637e214" containerName="maas-api" containerID="cri-o://ea3b004fed94f7566f962ec9ac5690df8996ddcad7988a442145c5b22f48dccc" gracePeriod=30 Oct 30 19:40:05.691669 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.691640 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:40:05.747715 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.747678 5047 generic.go:356] "Generic (PLEG): container finished" podID="eca5e1e9-76eb-4731-8041-1f521637e214" containerID="ea3b004fed94f7566f962ec9ac5690df8996ddcad7988a442145c5b22f48dccc" exitCode=0 Oct 30 19:40:05.747931 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.747725 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" Oct 30 19:40:05.747931 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.747762 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" event={"ID":"eca5e1e9-76eb-4731-8041-1f521637e214","Type":"ContainerDied","Data":"ea3b004fed94f7566f962ec9ac5690df8996ddcad7988a442145c5b22f48dccc"} Oct 30 19:40:05.747931 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.747834 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9" event={"ID":"eca5e1e9-76eb-4731-8041-1f521637e214","Type":"ContainerDied","Data":"3c4c5c183da5c86d77474a183e45867fc0a0c86558ca9a4d11a7afc5493afcc0"} Oct 30 19:40:05.747931 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.747868 5047 scope.go:122] "RemoveContainer" containerID="ea3b004fed94f7566f962ec9ac5690df8996ddcad7988a442145c5b22f48dccc" Oct 30 19:40:05.763864 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.763830 5047 scope.go:122] "RemoveContainer" containerID="ea3b004fed94f7566f962ec9ac5690df8996ddcad7988a442145c5b22f48dccc" Oct 30 19:40:05.764332 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:40:05.764299 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea3b004fed94f7566f962ec9ac5690df8996ddcad7988a442145c5b22f48dccc\": container with ID starting with ea3b004fed94f7566f962ec9ac5690df8996ddcad7988a442145c5b22f48dccc not found: ID does not exist" containerID="ea3b004fed94f7566f962ec9ac5690df8996ddcad7988a442145c5b22f48dccc" Oct 30 19:40:05.764468 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.764344 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea3b004fed94f7566f962ec9ac5690df8996ddcad7988a442145c5b22f48dccc"} err="failed to get container status \"ea3b004fed94f7566f962ec9ac5690df8996ddcad7988a442145c5b22f48dccc\": rpc error: code = NotFound desc = could not find container \"ea3b004fed94f7566f962ec9ac5690df8996ddcad7988a442145c5b22f48dccc\": container with ID starting with ea3b004fed94f7566f962ec9ac5690df8996ddcad7988a442145c5b22f48dccc not found: ID does not exist" Oct 30 19:40:05.836708 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.836675 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eca5e1e9-76eb-4731-8041-1f521637e214-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eca5e1e9-76eb-4731-8041-1f521637e214-openshift-service-ca\") pod \"eca5e1e9-76eb-4731-8041-1f521637e214\" (UID: \"eca5e1e9-76eb-4731-8041-1f521637e214\") " Oct 30 19:40:05.836922 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.836789 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca5e1e9-76eb-4731-8041-1f521637e214-kube-api-access-975jz\" (UniqueName: \"kubernetes.io/projected/eca5e1e9-76eb-4731-8041-1f521637e214-kube-api-access-975jz\") pod \"eca5e1e9-76eb-4731-8041-1f521637e214\" (UID: \"eca5e1e9-76eb-4731-8041-1f521637e214\") " Oct 30 19:40:05.836922 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.836879 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eca5e1e9-76eb-4731-8041-1f521637e214-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eca5e1e9-76eb-4731-8041-1f521637e214-metrics-certs\") pod \"eca5e1e9-76eb-4731-8041-1f521637e214\" (UID: \"eca5e1e9-76eb-4731-8041-1f521637e214\") " Oct 30 19:40:05.837065 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.836955 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eca5e1e9-76eb-4731-8041-1f521637e214-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eca5e1e9-76eb-4731-8041-1f521637e214-maas-api-tls\") pod \"eca5e1e9-76eb-4731-8041-1f521637e214\" (UID: \"eca5e1e9-76eb-4731-8041-1f521637e214\") " Oct 30 19:40:05.837200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.837168 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca5e1e9-76eb-4731-8041-1f521637e214-openshift-service-ca" pod "eca5e1e9-76eb-4731-8041-1f521637e214" (UID: "eca5e1e9-76eb-4731-8041-1f521637e214"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:05.838117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.838091 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eca5e1e9-76eb-4731-8041-1f521637e214-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:05.839596 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.839567 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca5e1e9-76eb-4731-8041-1f521637e214-metrics-certs" pod "eca5e1e9-76eb-4731-8041-1f521637e214" (UID: "eca5e1e9-76eb-4731-8041-1f521637e214"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:40:05.839983 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.839956 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca5e1e9-76eb-4731-8041-1f521637e214-kube-api-access-975jz" pod "eca5e1e9-76eb-4731-8041-1f521637e214" (UID: "eca5e1e9-76eb-4731-8041-1f521637e214"). InnerVolumeSpecName "kube-api-access-975jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:05.840183 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.840160 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca5e1e9-76eb-4731-8041-1f521637e214-maas-api-tls" pod "eca5e1e9-76eb-4731-8041-1f521637e214" (UID: "eca5e1e9-76eb-4731-8041-1f521637e214"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:40:05.964705 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.964604 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eca5e1e9-76eb-4731-8041-1f521637e214-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:05.964705 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.964649 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eca5e1e9-76eb-4731-8041-1f521637e214-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:05.964705 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:05.964667 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-975jz\" (UniqueName: \"kubernetes.io/projected/eca5e1e9-76eb-4731-8041-1f521637e214-kube-api-access-975jz\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:06.260288 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:06.260191 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9"] Oct 30 19:40:06.271393 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:06.271349 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4cb13-65fb7749-htkc9"] Oct 30 19:40:07.053729 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:07.053694 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eca5e1e9-76eb-4731-8041-1f521637e214" path="/var/lib/kubelet/pods/eca5e1e9-76eb-4731-8041-1f521637e214/volumes" Oct 30 19:40:08.395797 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:08.395759 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc"] Oct 30 19:40:09.607277 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:40:09.607246 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:40:12.812031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:12.811122 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:40:12.822804 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:12.822760 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:40:12.903642 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:12.903597 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" podUID="68ea348d-f2bf-4ee5-9062-9649b71be4c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.140:8000/ready\": context deadline exceeded" Oct 30 19:40:12.964363 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:12.964315 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:40:13.220073 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:13.180147 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" podUID="6dbadedf-fc49-43d8-82af-d7024864cc1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.146:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:40:13.220073 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:13.180960 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" podUID="7456b22a-9c80-4fbb-9236-08eed4067902" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:40:13.363227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:13.363177 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:40:13.364077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:13.363655 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:40:13.611031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:13.596095 5047 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:40:13.828043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:13.817105 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:40:13.828043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:13.817727 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:40:13.830101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:13.829100 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 19:40:13.830101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:13.829690 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:40:13.914869 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:13.914729 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" podUID="68ea348d-f2bf-4ee5-9062-9649b71be4c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.140:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:40:14.103432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:14.002542 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:40:14.534038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:14.113742 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:40:14.534038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:14.514773 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:40:14.537944 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:14.186322 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" podUID="7456b22a-9c80-4fbb-9236-08eed4067902" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:40:14.552293 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:14.395827 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 19:40:14.566258 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:14.186360 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" podUID="6dbadedf-fc49-43d8-82af-d7024864cc1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.146:8000/ready\": context deadline exceeded" Oct 30 19:40:14.594045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:14.565410 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:40:14.649492 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:14.649455 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:40:14.859457 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:14.670318 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:40:14.859457 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:14.824070 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:40:14.859457 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:14.836493 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:40:15.129094 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:15.128479 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" podUID="68ea348d-f2bf-4ee5-9062-9649b71be4c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.140:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:40:15.312043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:15.311830 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:40:15.312043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:15.311907 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:40:15.319040 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:15.317638 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:40:15.327455 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:15.327421 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:40:15.336036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:15.332982 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 19:40:15.401070 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:15.400954 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:40:15.431388 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:15.431262 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 19:40:15.505325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:15.484300 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 19:40:18.371025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:18.369274 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6"] Oct 30 19:40:18.371025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:18.369534 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" podUID="a4f73d85-9e6c-474c-907c-f1ae40bdcbda" containerName="istio-proxy" containerID="cri-o://5a18d6cb3e81fc47981e995056919b0a483017c7f512becda5f7c3a7163e3308" gracePeriod=30 Oct 30 19:40:18.377999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:18.377964 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq"] Oct 30 19:40:18.378474 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:18.378437 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" podUID="7abf871e-98f2-4f49-9660-4e5640c12a1a" containerName="istio-proxy" containerID="cri-o://7c57e8e496e4930ced53cb41f5d0e33ebadfd6d82d11ecacd30d0c320b4edcac" gracePeriod=30 Oct 30 19:40:23.421214 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.421162 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" podUID="68ea348d-f2bf-4ee5-9062-9649b71be4c3" containerName="main" containerID="cri-o://9577a0d4bd3235a7403ae676b856845ab28263d5c81a0f87a549f80e12a238ac" gracePeriod=45 Oct 30 19:40:23.541148 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.541119 5047 generic.go:356] "Generic (PLEG): container finished" podID="a4f73d85-9e6c-474c-907c-f1ae40bdcbda" containerID="5a18d6cb3e81fc47981e995056919b0a483017c7f512becda5f7c3a7163e3308" exitCode=0 Oct 30 19:40:23.541248 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.541195 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" event={"ID":"a4f73d85-9e6c-474c-907c-f1ae40bdcbda","Type":"ContainerDied","Data":"5a18d6cb3e81fc47981e995056919b0a483017c7f512becda5f7c3a7163e3308"} Oct 30 19:40:23.543485 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.543447 5047 generic.go:356] "Generic (PLEG): container finished" podID="7abf871e-98f2-4f49-9660-4e5640c12a1a" containerID="7c57e8e496e4930ced53cb41f5d0e33ebadfd6d82d11ecacd30d0c320b4edcac" exitCode=0 Oct 30 19:40:23.543562 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.543528 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" event={"ID":"7abf871e-98f2-4f49-9660-4e5640c12a1a","Type":"ContainerDied","Data":"7c57e8e496e4930ced53cb41f5d0e33ebadfd6d82d11ecacd30d0c320b4edcac"} Oct 30 19:40:23.545791 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.545771 5047 generic.go:356] "Generic (PLEG): container finished" podID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" containerID="69456a855408608c1513822806b145329f66d796c7f1281804f4aec66378faab" exitCode=0 Oct 30 19:40:23.545906 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.545849 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" event={"ID":"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c","Type":"ContainerDied","Data":"69456a855408608c1513822806b145329f66d796c7f1281804f4aec66378faab"} Oct 30 19:40:23.546341 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.546322 5047 scope.go:122] "RemoveContainer" containerID="69456a855408608c1513822806b145329f66d796c7f1281804f4aec66378faab" Oct 30 19:40:23.548023 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.547989 5047 generic.go:356] "Generic (PLEG): container finished" podID="68ea348d-f2bf-4ee5-9062-9649b71be4c3" containerID="9577a0d4bd3235a7403ae676b856845ab28263d5c81a0f87a549f80e12a238ac" exitCode=0 Oct 30 19:40:23.548128 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.548066 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" event={"ID":"68ea348d-f2bf-4ee5-9062-9649b71be4c3","Type":"ContainerDied","Data":"9577a0d4bd3235a7403ae676b856845ab28263d5c81a0f87a549f80e12a238ac"} Oct 30 19:40:23.548462 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.548434 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:40:23.548534 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.548499 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:40:23.548534 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.548532 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:40:23.676717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.676693 5047 patch_prober.go:29] interesting pod/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:40:23.676889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.676768 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" podUID="7abf871e-98f2-4f49-9660-4e5640c12a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:40:23.759598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.759563 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:40:23.820076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.815755 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:40:23.826509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.824668 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istiod-ca-cert\") pod \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " Oct 30 19:40:23.826509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.824855 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-credential-socket\") pod \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " Oct 30 19:40:23.826509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.824939 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-podinfo\") pod \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " Oct 30 19:40:23.826509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.824987 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-workload-socket\") pod \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " Oct 30 19:40:23.826509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.825106 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-workload-certs\") pod \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " Oct 30 19:40:23.826509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.825189 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-kube-api-access-qcsfj\" (UniqueName: \"kubernetes.io/projected/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-kube-api-access-qcsfj\") pod \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " Oct 30 19:40:23.826509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.825233 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-data\") pod \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " Oct 30 19:40:23.826509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.825472 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-token\" (UniqueName: \"kubernetes.io/projected/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-token\") pod \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " Oct 30 19:40:23.826509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.825540 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-envoy\") pod \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\" (UID: \"a4f73d85-9e6c-474c-907c-f1ae40bdcbda\") " Oct 30 19:40:23.826509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.826280 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-workload-socket" pod "a4f73d85-9e6c-474c-907c-f1ae40bdcbda" (UID: "a4f73d85-9e6c-474c-907c-f1ae40bdcbda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:23.827517 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.827293 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-credential-socket" pod "a4f73d85-9e6c-474c-907c-f1ae40bdcbda" (UID: "a4f73d85-9e6c-474c-907c-f1ae40bdcbda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:23.827517 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.827309 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istiod-ca-cert" pod "a4f73d85-9e6c-474c-907c-f1ae40bdcbda" (UID: "a4f73d85-9e6c-474c-907c-f1ae40bdcbda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:23.827694 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.827631 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-workload-certs" pod "a4f73d85-9e6c-474c-907c-f1ae40bdcbda" (UID: "a4f73d85-9e6c-474c-907c-f1ae40bdcbda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:23.827860 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.827779 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-data" pod "a4f73d85-9e6c-474c-907c-f1ae40bdcbda" (UID: "a4f73d85-9e6c-474c-907c-f1ae40bdcbda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:23.828399 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.828374 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:23.828493 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.828404 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:23.828493 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.828419 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:23.828493 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.828434 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:23.828493 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.828447 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:23.831069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.830996 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-token" pod "a4f73d85-9e6c-474c-907c-f1ae40bdcbda" (UID: "a4f73d85-9e6c-474c-907c-f1ae40bdcbda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:23.831625 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.831595 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-kube-api-access-qcsfj" pod "a4f73d85-9e6c-474c-907c-f1ae40bdcbda" (UID: "a4f73d85-9e6c-474c-907c-f1ae40bdcbda"). InnerVolumeSpecName "kube-api-access-qcsfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:23.831843 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.831803 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-podinfo" pod "a4f73d85-9e6c-474c-907c-f1ae40bdcbda" (UID: "a4f73d85-9e6c-474c-907c-f1ae40bdcbda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:40:23.832318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.832295 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-envoy" pod "a4f73d85-9e6c-474c-907c-f1ae40bdcbda" (UID: "a4f73d85-9e6c-474c-907c-f1ae40bdcbda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:23.886531 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.886500 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:40:23.936216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.931871 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-home\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-home\") pod \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " Oct 30 19:40:23.936216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.931973 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-model-cache\") pod \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " Oct 30 19:40:23.936216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.932081 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-tmp-dir\") pod \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " Oct 30 19:40:23.936216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.932237 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-dshm\") pod \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " Oct 30 19:40:23.936216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.932383 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68ea348d-f2bf-4ee5-9062-9649b71be4c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/68ea348d-f2bf-4ee5-9062-9649b71be4c3-tls-certs\") pod \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " Oct 30 19:40:23.936216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.932537 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ea348d-f2bf-4ee5-9062-9649b71be4c3-kube-api-access-4vkts\" (UniqueName: \"kubernetes.io/projected/68ea348d-f2bf-4ee5-9062-9649b71be4c3-kube-api-access-4vkts\") pod \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\" (UID: \"68ea348d-f2bf-4ee5-9062-9649b71be4c3\") " Oct 30 19:40:23.936216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.933904 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcsfj\" (UniqueName: \"kubernetes.io/projected/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-kube-api-access-qcsfj\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:23.936216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.933938 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:23.936216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.933951 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:23.936216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.933965 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f73d85-9e6c-474c-907c-f1ae40bdcbda-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:23.936216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.934403 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-home" pod "68ea348d-f2bf-4ee5-9062-9649b71be4c3" (UID: "68ea348d-f2bf-4ee5-9062-9649b71be4c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:23.936216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.934593 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-model-cache" pod "68ea348d-f2bf-4ee5-9062-9649b71be4c3" (UID: "68ea348d-f2bf-4ee5-9062-9649b71be4c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:23.936216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.934768 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-tmp-dir" pod "68ea348d-f2bf-4ee5-9062-9649b71be4c3" (UID: "68ea348d-f2bf-4ee5-9062-9649b71be4c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:23.939588 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.939555 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-dshm" pod "68ea348d-f2bf-4ee5-9062-9649b71be4c3" (UID: "68ea348d-f2bf-4ee5-9062-9649b71be4c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:23.948056 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.942643 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ea348d-f2bf-4ee5-9062-9649b71be4c3-tls-certs" pod "68ea348d-f2bf-4ee5-9062-9649b71be4c3" (UID: "68ea348d-f2bf-4ee5-9062-9649b71be4c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:40:23.953658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:23.953613 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ea348d-f2bf-4ee5-9062-9649b71be4c3-kube-api-access-4vkts" pod "68ea348d-f2bf-4ee5-9062-9649b71be4c3" (UID: "68ea348d-f2bf-4ee5-9062-9649b71be4c3"). InnerVolumeSpecName "kube-api-access-4vkts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:24.042480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.040062 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vkts\" (UniqueName: \"kubernetes.io/projected/68ea348d-f2bf-4ee5-9062-9649b71be4c3-kube-api-access-4vkts\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:24.042480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.040101 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:24.042480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.040116 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:24.042480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.040131 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:24.042480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.040144 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68ea348d-f2bf-4ee5-9062-9649b71be4c3-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:24.042480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.040156 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68ea348d-f2bf-4ee5-9062-9649b71be4c3-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:24.152135 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.152106 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:40:24.243238 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.243148 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-envoy\") pod \"7abf871e-98f2-4f49-9660-4e5640c12a1a\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " Oct 30 19:40:24.243539 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.243509 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-workload-socket\") pod \"7abf871e-98f2-4f49-9660-4e5640c12a1a\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " Oct 30 19:40:24.245167 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.243802 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-workload-socket" pod "7abf871e-98f2-4f49-9660-4e5640c12a1a" (UID: "7abf871e-98f2-4f49-9660-4e5640c12a1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:24.245279 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.245259 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-token\" (UniqueName: \"kubernetes.io/projected/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-token\") pod \"7abf871e-98f2-4f49-9660-4e5640c12a1a\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " Oct 30 19:40:24.245365 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.245351 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-workload-certs\") pod \"7abf871e-98f2-4f49-9660-4e5640c12a1a\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " Oct 30 19:40:24.245440 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.245382 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-podinfo\") pod \"7abf871e-98f2-4f49-9660-4e5640c12a1a\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " Oct 30 19:40:24.245491 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.245452 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-data\") pod \"7abf871e-98f2-4f49-9660-4e5640c12a1a\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " Oct 30 19:40:24.245631 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.245614 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7abf871e-98f2-4f49-9660-4e5640c12a1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7abf871e-98f2-4f49-9660-4e5640c12a1a-istiod-ca-cert\") pod \"7abf871e-98f2-4f49-9660-4e5640c12a1a\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " Oct 30 19:40:24.245694 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.245677 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7abf871e-98f2-4f49-9660-4e5640c12a1a-kube-api-access-fm7rk\" (UniqueName: \"kubernetes.io/projected/7abf871e-98f2-4f49-9660-4e5640c12a1a-kube-api-access-fm7rk\") pod \"7abf871e-98f2-4f49-9660-4e5640c12a1a\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " Oct 30 19:40:24.245789 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.245773 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-credential-socket\") pod \"7abf871e-98f2-4f49-9660-4e5640c12a1a\" (UID: \"7abf871e-98f2-4f49-9660-4e5640c12a1a\") " Oct 30 19:40:24.245860 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.245786 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-envoy" pod "7abf871e-98f2-4f49-9660-4e5640c12a1a" (UID: "7abf871e-98f2-4f49-9660-4e5640c12a1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:24.246199 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.246177 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abf871e-98f2-4f49-9660-4e5640c12a1a-istiod-ca-cert" pod "7abf871e-98f2-4f49-9660-4e5640c12a1a" (UID: "7abf871e-98f2-4f49-9660-4e5640c12a1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:24.246392 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.246360 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-workload-certs" pod "7abf871e-98f2-4f49-9660-4e5640c12a1a" (UID: "7abf871e-98f2-4f49-9660-4e5640c12a1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:24.246453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.246402 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-credential-socket" pod "7abf871e-98f2-4f49-9660-4e5640c12a1a" (UID: "7abf871e-98f2-4f49-9660-4e5640c12a1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:24.246570 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.246551 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-data" pod "7abf871e-98f2-4f49-9660-4e5640c12a1a" (UID: "7abf871e-98f2-4f49-9660-4e5640c12a1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:24.246818 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.246803 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:24.246873 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.246822 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:24.246873 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.246835 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7abf871e-98f2-4f49-9660-4e5640c12a1a-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:24.246873 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.246846 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:24.246873 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.246857 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:24.246873 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.246868 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf871e-98f2-4f49-9660-4e5640c12a1a-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:24.248957 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.248932 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-token" pod "7abf871e-98f2-4f49-9660-4e5640c12a1a" (UID: "7abf871e-98f2-4f49-9660-4e5640c12a1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:24.249118 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.248933 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-podinfo" pod "7abf871e-98f2-4f49-9660-4e5640c12a1a" (UID: "7abf871e-98f2-4f49-9660-4e5640c12a1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:40:24.249118 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.248964 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abf871e-98f2-4f49-9660-4e5640c12a1a-kube-api-access-fm7rk" pod "7abf871e-98f2-4f49-9660-4e5640c12a1a" (UID: "7abf871e-98f2-4f49-9660-4e5640c12a1a"). InnerVolumeSpecName "kube-api-access-fm7rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:24.347980 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.347944 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:24.347980 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.347979 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7abf871e-98f2-4f49-9660-4e5640c12a1a-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:24.348254 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.347993 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm7rk\" (UniqueName: \"kubernetes.io/projected/7abf871e-98f2-4f49-9660-4e5640c12a1a-kube-api-access-fm7rk\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:24.586155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.585144 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" event={"ID":"a4f73d85-9e6c-474c-907c-f1ae40bdcbda","Type":"ContainerDied","Data":"2529ed4c70061d703a9c4d565bcb9cc879a72f1f4251e38998993c5007aac56e"} Oct 30 19:40:24.586155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.585242 5047 scope.go:122] "RemoveContainer" containerID="5a18d6cb3e81fc47981e995056919b0a483017c7f512becda5f7c3a7163e3308" Oct 30 19:40:24.586155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.585702 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6" Oct 30 19:40:24.594838 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.593468 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" event={"ID":"7abf871e-98f2-4f49-9660-4e5640c12a1a","Type":"ContainerDied","Data":"47bf179e9aee8ce61e76c6856f235bcceec3bb48d833c79d31d0399428d6eb74"} Oct 30 19:40:24.594838 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.593561 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq" Oct 30 19:40:24.606028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.597950 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" event={"ID":"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c","Type":"ContainerStarted","Data":"f106eca16669eaa9a8e6b80c70630c80179e56c6eefdf382c212cb4cd09848c5"} Oct 30 19:40:24.606028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.603284 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" event={"ID":"68ea348d-f2bf-4ee5-9062-9649b71be4c3","Type":"ContainerDied","Data":"6b81092ee7c7eaaeb7cb1f555ed9d077efc9e373a0c45ff6a72421ebf75fd647"} Oct 30 19:40:24.606028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.603401 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc" Oct 30 19:40:24.630227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.630193 5047 scope.go:122] "RemoveContainer" containerID="7c57e8e496e4930ced53cb41f5d0e33ebadfd6d82d11ecacd30d0c320b4edcac" Oct 30 19:40:24.674770 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.674735 5047 scope.go:122] "RemoveContainer" containerID="9577a0d4bd3235a7403ae676b856845ab28263d5c81a0f87a549f80e12a238ac" Oct 30 19:40:24.707642 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.707605 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc"] Oct 30 19:40:24.711819 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.711789 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f4ab4a/test-model-f4ab4a-kserve-64fcbc75f9-fdkgc"] Oct 30 19:40:24.767193 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.767151 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq"] Oct 30 19:40:24.771495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.771468 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-dwfxq"] Oct 30 19:40:24.788351 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.788317 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6"] Oct 30 19:40:24.792306 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.792281 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6e0c55-openshift-default-88b6596b5-hd6d6"] Oct 30 19:40:24.897157 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.897061 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:40:24.898868 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.898839 5047 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:40:24.898997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:24.898898 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" podUID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 19:40:25.067352 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:25.067308 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68ea348d-f2bf-4ee5-9062-9649b71be4c3" path="/var/lib/kubelet/pods/68ea348d-f2bf-4ee5-9062-9649b71be4c3/volumes" Oct 30 19:40:25.068020 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:25.067981 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7abf871e-98f2-4f49-9660-4e5640c12a1a" path="/var/lib/kubelet/pods/7abf871e-98f2-4f49-9660-4e5640c12a1a/volumes" Oct 30 19:40:25.068590 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:25.068562 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4f73d85-9e6c-474c-907c-f1ae40bdcbda" path="/var/lib/kubelet/pods/a4f73d85-9e6c-474c-907c-f1ae40bdcbda/volumes" Oct 30 19:40:25.897620 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:25.897586 5047 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:40:25.898072 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:25.897642 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" podUID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 19:40:26.897507 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:26.897474 5047 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:40:26.897687 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:26.897528 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" podUID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 19:40:27.897404 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:27.897361 5047 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:40:27.897576 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:27.897444 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" podUID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 19:40:28.897695 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:28.897656 5047 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:40:28.898266 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:28.897734 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" podUID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 19:40:29.899561 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:29.897395 5047 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:40:29.899561 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:29.897458 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" podUID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 19:40:30.902900 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:30.902866 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:40:30.903364 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:30.903344 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:40:30.904267 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:30.904251 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:40:33.364479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:33.364426 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd"] Oct 30 19:40:33.365048 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:33.364750 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" podUID="90f0e672-c84b-4721-9a09-363ffdf05947" containerName="istio-proxy" containerID="cri-o://a44b94fa2a2c7c29852894542ad33890811a80f30951ab50a8e21b0185f7618e" gracePeriod=30 Oct 30 19:40:38.716209 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.716177 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:40:38.818211 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.818179 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-workload-certs\") pod \"90f0e672-c84b-4721-9a09-363ffdf05947\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " Oct 30 19:40:38.818427 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.818255 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90f0e672-c84b-4721-9a09-363ffdf05947-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f0e672-c84b-4721-9a09-363ffdf05947-istio-podinfo\") pod \"90f0e672-c84b-4721-9a09-363ffdf05947\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " Oct 30 19:40:38.818427 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.818334 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-istio-envoy\") pod \"90f0e672-c84b-4721-9a09-363ffdf05947\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " Oct 30 19:40:38.818427 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.818389 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f0e672-c84b-4721-9a09-363ffdf05947-istio-token\" (UniqueName: \"kubernetes.io/projected/90f0e672-c84b-4721-9a09-363ffdf05947-istio-token\") pod \"90f0e672-c84b-4721-9a09-363ffdf05947\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " Oct 30 19:40:38.818593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.818497 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-istio-data\") pod \"90f0e672-c84b-4721-9a09-363ffdf05947\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " Oct 30 19:40:38.818593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.818524 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-workload-certs" pod "90f0e672-c84b-4721-9a09-363ffdf05947" (UID: "90f0e672-c84b-4721-9a09-363ffdf05947"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:38.818699 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.818670 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f0e672-c84b-4721-9a09-363ffdf05947-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f0e672-c84b-4721-9a09-363ffdf05947-istiod-ca-cert\") pod \"90f0e672-c84b-4721-9a09-363ffdf05947\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " Oct 30 19:40:38.818769 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.818751 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f0e672-c84b-4721-9a09-363ffdf05947-kube-api-access-mkjxp\" (UniqueName: \"kubernetes.io/projected/90f0e672-c84b-4721-9a09-363ffdf05947-kube-api-access-mkjxp\") pod \"90f0e672-c84b-4721-9a09-363ffdf05947\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " Oct 30 19:40:38.818841 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.818825 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-credential-socket\") pod \"90f0e672-c84b-4721-9a09-363ffdf05947\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " Oct 30 19:40:38.818895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.818835 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-istio-data" pod "90f0e672-c84b-4721-9a09-363ffdf05947" (UID: "90f0e672-c84b-4721-9a09-363ffdf05947"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:38.818895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.818865 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-workload-socket\") pod \"90f0e672-c84b-4721-9a09-363ffdf05947\" (UID: \"90f0e672-c84b-4721-9a09-363ffdf05947\") " Oct 30 19:40:38.819137 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.819109 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f0e672-c84b-4721-9a09-363ffdf05947-istiod-ca-cert" pod "90f0e672-c84b-4721-9a09-363ffdf05947" (UID: "90f0e672-c84b-4721-9a09-363ffdf05947"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:38.819379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.819359 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-credential-socket" pod "90f0e672-c84b-4721-9a09-363ffdf05947" (UID: "90f0e672-c84b-4721-9a09-363ffdf05947"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:38.819556 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.819536 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-workload-socket" pod "90f0e672-c84b-4721-9a09-363ffdf05947" (UID: "90f0e672-c84b-4721-9a09-363ffdf05947"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:38.819731 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.819714 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:38.819805 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.819733 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f0e672-c84b-4721-9a09-363ffdf05947-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:38.819805 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.819742 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:38.819805 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.819753 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:38.819805 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.819762 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:38.821160 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.820996 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f0e672-c84b-4721-9a09-363ffdf05947-kube-api-access-mkjxp" pod "90f0e672-c84b-4721-9a09-363ffdf05947" (UID: "90f0e672-c84b-4721-9a09-363ffdf05947"). InnerVolumeSpecName "kube-api-access-mkjxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:38.821160 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.821133 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-istio-envoy" pod "90f0e672-c84b-4721-9a09-363ffdf05947" (UID: "90f0e672-c84b-4721-9a09-363ffdf05947"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:38.821863 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.821826 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90f0e672-c84b-4721-9a09-363ffdf05947-istio-podinfo" pod "90f0e672-c84b-4721-9a09-363ffdf05947" (UID: "90f0e672-c84b-4721-9a09-363ffdf05947"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:40:38.823478 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.823452 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f0e672-c84b-4721-9a09-363ffdf05947-istio-token" pod "90f0e672-c84b-4721-9a09-363ffdf05947" (UID: "90f0e672-c84b-4721-9a09-363ffdf05947"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:38.823478 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.823456 5047 generic.go:356] "Generic (PLEG): container finished" podID="90f0e672-c84b-4721-9a09-363ffdf05947" containerID="a44b94fa2a2c7c29852894542ad33890811a80f30951ab50a8e21b0185f7618e" exitCode=0 Oct 30 19:40:38.823658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.823480 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" event={"ID":"90f0e672-c84b-4721-9a09-363ffdf05947","Type":"ContainerDied","Data":"a44b94fa2a2c7c29852894542ad33890811a80f30951ab50a8e21b0185f7618e"} Oct 30 19:40:38.823658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.823523 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" event={"ID":"90f0e672-c84b-4721-9a09-363ffdf05947","Type":"ContainerDied","Data":"80ee50a156b2daf9e45d1c61e55a1f981deb9ea137474fa153bd70bd0eb7493c"} Oct 30 19:40:38.823658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.823523 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd" Oct 30 19:40:38.823658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.823597 5047 scope.go:122] "RemoveContainer" containerID="a44b94fa2a2c7c29852894542ad33890811a80f30951ab50a8e21b0185f7618e" Oct 30 19:40:38.946400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.929125 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f0e672-c84b-4721-9a09-363ffdf05947-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:38.946400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.929153 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f0e672-c84b-4721-9a09-363ffdf05947-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:38.946400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.929167 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkjxp\" (UniqueName: \"kubernetes.io/projected/90f0e672-c84b-4721-9a09-363ffdf05947-kube-api-access-mkjxp\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:38.946400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.929182 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f0e672-c84b-4721-9a09-363ffdf05947-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:40:38.958949 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.958870 5047 scope.go:122] "RemoveContainer" containerID="b1250f61e90a56e35aa35cdae53794312ff17d973da85d4631f4b2a7abad7105" Oct 30 19:40:38.993825 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.993777 5047 scope.go:122] "RemoveContainer" containerID="a44b94fa2a2c7c29852894542ad33890811a80f30951ab50a8e21b0185f7618e" Oct 30 19:40:38.994284 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:40:38.994255 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a44b94fa2a2c7c29852894542ad33890811a80f30951ab50a8e21b0185f7618e\": container with ID starting with a44b94fa2a2c7c29852894542ad33890811a80f30951ab50a8e21b0185f7618e not found: ID does not exist" containerID="a44b94fa2a2c7c29852894542ad33890811a80f30951ab50a8e21b0185f7618e" Oct 30 19:40:38.994359 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.994300 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44b94fa2a2c7c29852894542ad33890811a80f30951ab50a8e21b0185f7618e"} err="failed to get container status \"a44b94fa2a2c7c29852894542ad33890811a80f30951ab50a8e21b0185f7618e\": rpc error: code = NotFound desc = could not find container \"a44b94fa2a2c7c29852894542ad33890811a80f30951ab50a8e21b0185f7618e\": container with ID starting with a44b94fa2a2c7c29852894542ad33890811a80f30951ab50a8e21b0185f7618e not found: ID does not exist" Oct 30 19:40:38.994359 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.994323 5047 scope.go:122] "RemoveContainer" containerID="b1250f61e90a56e35aa35cdae53794312ff17d973da85d4631f4b2a7abad7105" Oct 30 19:40:38.994630 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:40:38.994608 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1250f61e90a56e35aa35cdae53794312ff17d973da85d4631f4b2a7abad7105\": container with ID starting with b1250f61e90a56e35aa35cdae53794312ff17d973da85d4631f4b2a7abad7105 not found: ID does not exist" containerID="b1250f61e90a56e35aa35cdae53794312ff17d973da85d4631f4b2a7abad7105" Oct 30 19:40:38.994698 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:38.994640 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1250f61e90a56e35aa35cdae53794312ff17d973da85d4631f4b2a7abad7105"} err="failed to get container status \"b1250f61e90a56e35aa35cdae53794312ff17d973da85d4631f4b2a7abad7105\": rpc error: code = NotFound desc = could not find container \"b1250f61e90a56e35aa35cdae53794312ff17d973da85d4631f4b2a7abad7105\": container with ID starting with b1250f61e90a56e35aa35cdae53794312ff17d973da85d4631f4b2a7abad7105 not found: ID does not exist" Oct 30 19:40:39.010120 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:39.009969 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd"] Oct 30 19:40:39.014585 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:39.014558 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d4cb13-openshift-default-84756765b7-8rvfd"] Oct 30 19:40:39.070538 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:39.070504 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f0e672-c84b-4721-9a09-363ffdf05947" path="/var/lib/kubelet/pods/90f0e672-c84b-4721-9a09-363ffdf05947/volumes" Oct 30 19:40:39.603426 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:40:39.603389 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:40:44.946331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:44.946294 5047 generic.go:356] "Generic (PLEG): container finished" podID="96bf353d-9dac-4c62-ac9e-c97d305d08a4" containerID="f484110505d1ebb6d86ee82833d9148eb10a96b87236e75a4f02ce47931e44de" exitCode=0 Oct 30 19:40:44.946817 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:44.946554 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" event={"ID":"96bf353d-9dac-4c62-ac9e-c97d305d08a4","Type":"ContainerDied","Data":"f484110505d1ebb6d86ee82833d9148eb10a96b87236e75a4f02ce47931e44de"} Oct 30 19:40:44.947168 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:44.947144 5047 scope.go:122] "RemoveContainer" containerID="f484110505d1ebb6d86ee82833d9148eb10a96b87236e75a4f02ce47931e44de" Oct 30 19:40:44.949284 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:44.949249 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:40:44.949373 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:44.949326 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:40:44.949373 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:44.949360 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:40:45.956835 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:45.956787 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" event={"ID":"96bf353d-9dac-4c62-ac9e-c97d305d08a4","Type":"ContainerStarted","Data":"b209e39dd0bd4727938fd9d590d76eb7b361e874d021f44fe2e239457c2aa2d6"} Oct 30 19:40:46.280152 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:46.280056 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:40:46.280336 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:46.280308 5047 patch_prober.go:29] interesting pod/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:40:46.280411 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:46.280368 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" podUID="96bf353d-9dac-4c62-ac9e-c97d305d08a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 19:40:47.117255 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.117213 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj"] Oct 30 19:40:47.118508 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.118482 5047 cpu_manager.go:414] "containerMap: removing container" podUID="90f0e672-c84b-4721-9a09-363ffdf05947" containerName="istio-proxy" Oct 30 19:40:47.118592 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.118509 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f0e672-c84b-4721-9a09-363ffdf05947" containerName="istio-proxy" Oct 30 19:40:47.118647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.118593 5047 cpu_manager.go:414] "containerMap: removing container" podUID="68ea348d-f2bf-4ee5-9062-9649b71be4c3" containerName="main" Oct 30 19:40:47.118647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.118604 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ea348d-f2bf-4ee5-9062-9649b71be4c3" containerName="main" Oct 30 19:40:47.118647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.118632 5047 cpu_manager.go:414] "containerMap: removing container" podUID="eca5e1e9-76eb-4731-8041-1f521637e214" containerName="maas-api" Oct 30 19:40:47.118647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.118641 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca5e1e9-76eb-4731-8041-1f521637e214" containerName="maas-api" Oct 30 19:40:47.118848 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.118659 5047 cpu_manager.go:414] "containerMap: removing container" podUID="a4f73d85-9e6c-474c-907c-f1ae40bdcbda" containerName="istio-proxy" Oct 30 19:40:47.118848 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.118669 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f73d85-9e6c-474c-907c-f1ae40bdcbda" containerName="istio-proxy" Oct 30 19:40:47.118848 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.118682 5047 cpu_manager.go:414] "containerMap: removing container" podUID="7abf871e-98f2-4f49-9660-4e5640c12a1a" containerName="istio-proxy" Oct 30 19:40:47.118848 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.118690 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abf871e-98f2-4f49-9660-4e5640c12a1a" containerName="istio-proxy" Oct 30 19:40:47.119069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.118905 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f0e672-c84b-4721-9a09-363ffdf05947" containerName="istio-proxy" Oct 30 19:40:47.119069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.118922 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ea348d-f2bf-4ee5-9062-9649b71be4c3" containerName="main" Oct 30 19:40:47.119069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.118934 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca5e1e9-76eb-4731-8041-1f521637e214" containerName="maas-api" Oct 30 19:40:47.119069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.118944 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f73d85-9e6c-474c-907c-f1ae40bdcbda" containerName="istio-proxy" Oct 30 19:40:47.119069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.118953 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="7abf871e-98f2-4f49-9660-4e5640c12a1a" containerName="istio-proxy" Oct 30 19:40:47.119069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.118968 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f0e672-c84b-4721-9a09-363ffdf05947" containerName="istio-proxy" Oct 30 19:40:47.122703 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.122676 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.159838 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.146817 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e3504b-openshift-default-dockercfg-ct8cl\"" Oct 30 19:40:47.183432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.183382 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj"] Oct 30 19:40:47.277242 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.277188 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-workload-certs\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.277462 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.277345 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299d2abb-3201-42ac-8f33-edc45040171a-istiod-ca-cert\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.277462 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.277384 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-credential-socket\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.277598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.277506 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-workload-socket\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.277598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.277539 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-istio-envoy\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.277712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.277661 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/299d2abb-3201-42ac-8f33-edc45040171a-istio-token\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.277775 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.277728 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-istio-data\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.277837 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.277810 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299d2abb-3201-42ac-8f33-edc45040171a-istio-podinfo\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.277894 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.277866 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbfkf\" (UniqueName: \"kubernetes.io/projected/299d2abb-3201-42ac-8f33-edc45040171a-kube-api-access-rbfkf\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.280109 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.280081 5047 patch_prober.go:29] interesting pod/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:40:47.280271 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.280140 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" podUID="96bf353d-9dac-4c62-ac9e-c97d305d08a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 19:40:47.387093 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.380282 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-workload-socket\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.387093 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.380360 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-istio-envoy\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.387093 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.380452 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/299d2abb-3201-42ac-8f33-edc45040171a-istio-token\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.387093 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.380529 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-istio-data\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.387093 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.380605 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299d2abb-3201-42ac-8f33-edc45040171a-istio-podinfo\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.387093 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.380691 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbfkf\" (UniqueName: \"kubernetes.io/projected/299d2abb-3201-42ac-8f33-edc45040171a-kube-api-access-rbfkf\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.387093 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.380761 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-workload-certs\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.387093 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.380862 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299d2abb-3201-42ac-8f33-edc45040171a-istiod-ca-cert\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.387093 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.380946 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-credential-socket\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.387093 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.381486 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-credential-socket\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.387093 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.381759 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-workload-socket\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.404032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.393548 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-workload-certs\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.404032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.394828 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299d2abb-3201-42ac-8f33-edc45040171a-istiod-ca-cert\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.409037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.407331 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-istio-envoy\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.443101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.430363 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299d2abb-3201-42ac-8f33-edc45040171a-istio-podinfo\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.443101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.432956 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-istio-data\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.443101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.436271 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbfkf\" (UniqueName: \"kubernetes.io/projected/299d2abb-3201-42ac-8f33-edc45040171a-kube-api-access-rbfkf\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.461701 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.447499 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/299d2abb-3201-42ac-8f33-edc45040171a-istio-token\") pod \"e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.489037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.488650 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:47.997117 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:40:47.997083 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299d2abb_3201_42ac_8f33_edc45040171a.slice/crio-8b178a3545fe843594f4967e589912d0245939febdb68981e5772aa82c21d734 WatchSource:0}: Error finding container 8b178a3545fe843594f4967e589912d0245939febdb68981e5772aa82c21d734: Status 404 returned error can't find the container with id 8b178a3545fe843594f4967e589912d0245939febdb68981e5772aa82c21d734 Oct 30 19:40:47.997665 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:47.997586 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj"] Oct 30 19:40:48.000088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:48.000053 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:40:48.000198 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:48.000138 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:40:48.000198 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:48.000185 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:40:48.286112 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:48.280106 5047 patch_prober.go:29] interesting pod/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:40:48.286112 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:48.280180 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" podUID="96bf353d-9dac-4c62-ac9e-c97d305d08a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 19:40:48.984946 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:48.984903 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" event={"ID":"299d2abb-3201-42ac-8f33-edc45040171a","Type":"ContainerStarted","Data":"000c8cc05ac04716cd74114a6bc3f4d2330f997caed5e8a45280bfb3001cbdb4"} Oct 30 19:40:48.985174 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:48.984969 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" event={"ID":"299d2abb-3201-42ac-8f33-edc45040171a","Type":"ContainerStarted","Data":"8b178a3545fe843594f4967e589912d0245939febdb68981e5772aa82c21d734"} Oct 30 19:40:49.064768 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:49.064690 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" podStartSLOduration=2.064666985 podStartE2EDuration="2.064666985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:49.031945561 +0000 UTC m=+2202.106400919" watchObservedRunningTime="2026-10-30 19:40:49.064666985 +0000 UTC m=+2202.139122344" Oct 30 19:40:49.283714 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:49.283615 5047 patch_prober.go:29] interesting pod/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:40:49.284027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:49.283962 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" podUID="96bf353d-9dac-4c62-ac9e-c97d305d08a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 19:40:49.489648 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:49.489610 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:49.496414 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:49.496378 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:49.993509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:49.993470 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:49.994533 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:49.994511 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:40:50.280245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:50.280135 5047 patch_prober.go:29] interesting pod/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:40:50.280245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:50.280225 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" podUID="96bf353d-9dac-4c62-ac9e-c97d305d08a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 19:40:50.762816 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:50.762771 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x"] Oct 30 19:40:50.791260 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:50.791221 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l"] Oct 30 19:40:51.280020 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:51.279970 5047 patch_prober.go:29] interesting pod/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:40:51.280214 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:51.280076 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" podUID="96bf353d-9dac-4c62-ac9e-c97d305d08a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 19:40:52.364192 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.364156 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:40:52.365476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.365397 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:40:52.366717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.366692 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:40:52.493054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.473978 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl"] Oct 30 19:40:52.493054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.475818 5047 cpu_manager.go:414] "containerMap: removing container" podUID="90f0e672-c84b-4721-9a09-363ffdf05947" containerName="istio-proxy" Oct 30 19:40:52.493054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.475855 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f0e672-c84b-4721-9a09-363ffdf05947" containerName="istio-proxy" Oct 30 19:40:52.506187 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.506152 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:40:52.540366 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.540285 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-469724-serving-cert\"" Oct 30 19:40:52.807085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.790200 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-maas-api-tls\") pod \"maas-api-e2e-inf-b-469724-59f488d86b-ghrgl\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:40:52.807085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.790336 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-metrics-certs\") pod \"maas-api-e2e-inf-b-469724-59f488d86b-ghrgl\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:40:52.807085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.790403 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-openshift-service-ca\") pod \"maas-api-e2e-inf-b-469724-59f488d86b-ghrgl\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:40:52.807085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.790460 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc6rz\" (UniqueName: \"kubernetes.io/projected/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-kube-api-access-hc6rz\") pod \"maas-api-e2e-inf-b-469724-59f488d86b-ghrgl\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:40:52.873139 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.873081 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl"] Oct 30 19:40:52.910380 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.910343 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-maas-api-tls\") pod \"maas-api-e2e-inf-b-469724-59f488d86b-ghrgl\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:40:52.910712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.910689 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-metrics-certs\") pod \"maas-api-e2e-inf-b-469724-59f488d86b-ghrgl\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:40:52.911472 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.911449 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-openshift-service-ca\") pod \"maas-api-e2e-inf-b-469724-59f488d86b-ghrgl\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:40:52.911847 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.911817 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc6rz\" (UniqueName: \"kubernetes.io/projected/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-kube-api-access-hc6rz\") pod \"maas-api-e2e-inf-b-469724-59f488d86b-ghrgl\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:40:52.913033 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:40:52.912990 5047 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-serving-cert: secret "maas-api-e2e-inf-b-469724-serving-cert" not found Oct 30 19:40:52.914785 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:40:52.914769 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-maas-api-tls podName:75e6cb8a-1ee5-4a80-a28f-e146bd24f649 nodeName:}" failed. No retries permitted until 2026-10-30 19:40:53.414745536 +0000 UTC m=+2206.489200887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-maas-api-tls") pod "maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" (UID: "75e6cb8a-1ee5-4a80-a28f-e146bd24f649") : secret "maas-api-e2e-inf-b-469724-serving-cert" not found Oct 30 19:40:52.915200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.914561 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-openshift-service-ca\") pod \"maas-api-e2e-inf-b-469724-59f488d86b-ghrgl\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:40:52.946066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:52.945860 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-metrics-certs\") pod \"maas-api-e2e-inf-b-469724-59f488d86b-ghrgl\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:40:53.050675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:53.010815 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc6rz\" (UniqueName: \"kubernetes.io/projected/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-kube-api-access-hc6rz\") pod \"maas-api-e2e-inf-b-469724-59f488d86b-ghrgl\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:40:53.460939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:53.460899 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-maas-api-tls\") pod \"maas-api-e2e-inf-b-469724-59f488d86b-ghrgl\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:40:53.461754 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:40:53.461643 5047 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-serving-cert: secret "maas-api-e2e-inf-b-469724-serving-cert" not found Oct 30 19:40:53.462048 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:40:53.462036 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-maas-api-tls podName:75e6cb8a-1ee5-4a80-a28f-e146bd24f649 nodeName:}" failed. No retries permitted until 2026-10-30 19:40:54.461977887 +0000 UTC m=+2207.536433232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-maas-api-tls") pod "maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" (UID: "75e6cb8a-1ee5-4a80-a28f-e146bd24f649") : secret "maas-api-e2e-inf-b-469724-serving-cert" not found Oct 30 19:40:54.574658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:54.574610 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-maas-api-tls\") pod \"maas-api-e2e-inf-b-469724-59f488d86b-ghrgl\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:40:54.620553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:54.620515 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-maas-api-tls\") pod \"maas-api-e2e-inf-b-469724-59f488d86b-ghrgl\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:40:54.820826 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:54.818641 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:40:55.190812 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:55.190777 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl"] Oct 30 19:40:55.191664 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:40:55.191635 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e6cb8a_1ee5_4a80_a28f_e146bd24f649.slice/crio-c0a7f8bc9532f386d0480ff6bb7037d2968ef381c4793e20dd05ef900e343a9a WatchSource:0}: Error finding container c0a7f8bc9532f386d0480ff6bb7037d2968ef381c4793e20dd05ef900e343a9a: Status 404 returned error can't find the container with id c0a7f8bc9532f386d0480ff6bb7037d2968ef381c4793e20dd05ef900e343a9a Oct 30 19:40:55.208738 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:55.208700 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" event={"ID":"75e6cb8a-1ee5-4a80-a28f-e146bd24f649","Type":"ContainerStarted","Data":"c0a7f8bc9532f386d0480ff6bb7037d2968ef381c4793e20dd05ef900e343a9a"} Oct 30 19:40:55.763958 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:55.763912 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" podUID="7d3adfad-8cca-4e98-9812-8c17c9cfb4c1" containerName="payload-processing" containerID="cri-o://c7e2592193c532f309801930c9dea7af8868a96f3f98a21263c029c1b579251a" gracePeriod=25 Oct 30 19:40:55.791614 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:55.791513 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" podUID="6a0824a6-857e-47a0-9bff-5b0ed8ce38e9" containerName="payload-pre-processing" containerID="cri-o://f49f3c9ec78854bc03d585c8a1255199dbe3b8e9e51e99213afe8e9ba2df949c" gracePeriod=25 Oct 30 19:40:56.218957 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:56.218922 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" event={"ID":"75e6cb8a-1ee5-4a80-a28f-e146bd24f649","Type":"ContainerStarted","Data":"92d8b2e1055a22f4fcb2af663590569960a9e99cc9b335aa729cd07cb3b57ff3"} Oct 30 19:40:56.219172 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:56.219001 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:40:56.254838 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:56.254780 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" podStartSLOduration=3.915515258 podStartE2EDuration="4.254765295s" totalImagesPullingTime="339.250037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:56.254072212 +0000 UTC m=+2209.328527574" watchObservedRunningTime="2026-10-30 19:40:56.254765295 +0000 UTC m=+2209.329220653" Oct 30 19:40:56.661763 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:56.661718 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp"] Oct 30 19:40:56.661965 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:56.661941 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" podUID="1d06c47f-e741-4465-a0bb-63b6544002da" containerName="istio-proxy" containerID="cri-o://578dee23a51f24b7f45024b7c7b561f8f253d4fd9163925b4cb0943a40aab0e9" gracePeriod=30 Oct 30 19:40:56.670163 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:56.670101 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk"] Oct 30 19:40:56.672302 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:56.670502 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" podUID="908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" containerName="istio-proxy" containerID="cri-o://0d81bdb27dc6388fc7151f31d572e32217f14a8614b8f06b880b4a6ce622bf54" gracePeriod=30 Oct 30 19:40:56.690627 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:56.690590 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs"] Oct 30 19:40:56.694037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:56.691283 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" podUID="a252f556-2d81-4164-8efa-65037946f382" containerName="istio-proxy" containerID="cri-o://1283e93829ab77a30c4b384bddfd704c8bc6c269937b20cc979c006cb3bd82bd" gracePeriod=30 Oct 30 19:40:59.143032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:59.142526 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6"] Oct 30 19:40:59.143032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:59.142799 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" podUID="d4d691a1-d91c-4985-9866-3a96f94b2e67" containerName="maas-api" containerID="cri-o://c2a1ae658f4e2776fee1c36b0225e171dfd2805f5bafe0364069599638439406" gracePeriod=30 Oct 30 19:40:59.977808 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:40:59.977764 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:41:00.046135 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.045988 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4d691a1-d91c-4985-9866-3a96f94b2e67-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d691a1-d91c-4985-9866-3a96f94b2e67-maas-api-tls\") pod \"d4d691a1-d91c-4985-9866-3a96f94b2e67\" (UID: \"d4d691a1-d91c-4985-9866-3a96f94b2e67\") " Oct 30 19:41:00.047939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.047903 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d691a1-d91c-4985-9866-3a96f94b2e67-kube-api-access-4jwbs\" (UniqueName: \"kubernetes.io/projected/d4d691a1-d91c-4985-9866-3a96f94b2e67-kube-api-access-4jwbs\") pod \"d4d691a1-d91c-4985-9866-3a96f94b2e67\" (UID: \"d4d691a1-d91c-4985-9866-3a96f94b2e67\") " Oct 30 19:41:00.048396 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.048368 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d691a1-d91c-4985-9866-3a96f94b2e67-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d691a1-d91c-4985-9866-3a96f94b2e67-openshift-service-ca\") pod \"d4d691a1-d91c-4985-9866-3a96f94b2e67\" (UID: \"d4d691a1-d91c-4985-9866-3a96f94b2e67\") " Oct 30 19:41:00.048527 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.048509 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4d691a1-d91c-4985-9866-3a96f94b2e67-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d691a1-d91c-4985-9866-3a96f94b2e67-metrics-certs\") pod \"d4d691a1-d91c-4985-9866-3a96f94b2e67\" (UID: \"d4d691a1-d91c-4985-9866-3a96f94b2e67\") " Oct 30 19:41:00.049855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.049824 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d691a1-d91c-4985-9866-3a96f94b2e67-openshift-service-ca" pod "d4d691a1-d91c-4985-9866-3a96f94b2e67" (UID: "d4d691a1-d91c-4985-9866-3a96f94b2e67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:00.051384 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.051349 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d691a1-d91c-4985-9866-3a96f94b2e67-kube-api-access-4jwbs" pod "d4d691a1-d91c-4985-9866-3a96f94b2e67" (UID: "d4d691a1-d91c-4985-9866-3a96f94b2e67"). InnerVolumeSpecName "kube-api-access-4jwbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:00.051506 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.051482 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d691a1-d91c-4985-9866-3a96f94b2e67-maas-api-tls" pod "d4d691a1-d91c-4985-9866-3a96f94b2e67" (UID: "d4d691a1-d91c-4985-9866-3a96f94b2e67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:41:00.052427 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.052401 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d691a1-d91c-4985-9866-3a96f94b2e67-metrics-certs" pod "d4d691a1-d91c-4985-9866-3a96f94b2e67" (UID: "d4d691a1-d91c-4985-9866-3a96f94b2e67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:41:00.185027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.184822 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d691a1-d91c-4985-9866-3a96f94b2e67-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:00.185027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.184869 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jwbs\" (UniqueName: \"kubernetes.io/projected/d4d691a1-d91c-4985-9866-3a96f94b2e67-kube-api-access-4jwbs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:00.185027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.184885 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d691a1-d91c-4985-9866-3a96f94b2e67-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:00.185027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.184902 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d691a1-d91c-4985-9866-3a96f94b2e67-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:00.332988 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.332941 5047 generic.go:356] "Generic (PLEG): container finished" podID="d4d691a1-d91c-4985-9866-3a96f94b2e67" containerID="c2a1ae658f4e2776fee1c36b0225e171dfd2805f5bafe0364069599638439406" exitCode=0 Oct 30 19:41:00.333257 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.333198 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" event={"ID":"d4d691a1-d91c-4985-9866-3a96f94b2e67","Type":"ContainerDied","Data":"c2a1ae658f4e2776fee1c36b0225e171dfd2805f5bafe0364069599638439406"} Oct 30 19:41:00.333257 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.333253 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" event={"ID":"d4d691a1-d91c-4985-9866-3a96f94b2e67","Type":"ContainerDied","Data":"3d929941246015b32f00811fc315456ce9ee02145bbdc6583ad6aa42a6a7b168"} Oct 30 19:41:00.333390 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.333283 5047 scope.go:122] "RemoveContainer" containerID="c2a1ae658f4e2776fee1c36b0225e171dfd2805f5bafe0364069599638439406" Oct 30 19:41:00.333628 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.333602 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6" Oct 30 19:41:00.388198 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.388154 5047 scope.go:122] "RemoveContainer" containerID="c2a1ae658f4e2776fee1c36b0225e171dfd2805f5bafe0364069599638439406" Oct 30 19:41:00.388641 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:00.388620 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a1ae658f4e2776fee1c36b0225e171dfd2805f5bafe0364069599638439406\": container with ID starting with c2a1ae658f4e2776fee1c36b0225e171dfd2805f5bafe0364069599638439406 not found: ID does not exist" containerID="c2a1ae658f4e2776fee1c36b0225e171dfd2805f5bafe0364069599638439406" Oct 30 19:41:00.388718 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.388652 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a1ae658f4e2776fee1c36b0225e171dfd2805f5bafe0364069599638439406"} err="failed to get container status \"c2a1ae658f4e2776fee1c36b0225e171dfd2805f5bafe0364069599638439406\": rpc error: code = NotFound desc = could not find container \"c2a1ae658f4e2776fee1c36b0225e171dfd2805f5bafe0364069599638439406\": container with ID starting with c2a1ae658f4e2776fee1c36b0225e171dfd2805f5bafe0364069599638439406 not found: ID does not exist" Oct 30 19:41:00.426334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.426299 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6"] Oct 30 19:41:00.432108 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:00.432076 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4ab4a-5b96f8c9f9-r2zg6"] Oct 30 19:41:01.062433 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:01.062390 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d691a1-d91c-4985-9866-3a96f94b2e67" path="/var/lib/kubelet/pods/d4d691a1-d91c-4985-9866-3a96f94b2e67/volumes" Oct 30 19:41:01.143156 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:01.143111 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" podUID="6a0824a6-857e-47a0-9bff-5b0ed8ce38e9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:41:02.108311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.108269 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:41:02.173709 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.173479 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" podUID="7d3adfad-8cca-4e98-9812-8c17c9cfb4c1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:41:02.186119 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.184897 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:41:02.259501 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.259456 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-istio-data\") pod \"a252f556-2d81-4164-8efa-65037946f382\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " Oct 30 19:41:02.259700 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.259524 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a252f556-2d81-4164-8efa-65037946f382-kube-api-access-8tqng\" (UniqueName: \"kubernetes.io/projected/a252f556-2d81-4164-8efa-65037946f382-kube-api-access-8tqng\") pod \"a252f556-2d81-4164-8efa-65037946f382\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " Oct 30 19:41:02.259700 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.259556 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a252f556-2d81-4164-8efa-65037946f382-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a252f556-2d81-4164-8efa-65037946f382-istio-podinfo\") pod \"a252f556-2d81-4164-8efa-65037946f382\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " Oct 30 19:41:02.259700 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.259591 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a252f556-2d81-4164-8efa-65037946f382-istio-token\" (UniqueName: \"kubernetes.io/projected/a252f556-2d81-4164-8efa-65037946f382-istio-token\") pod \"a252f556-2d81-4164-8efa-65037946f382\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " Oct 30 19:41:02.259700 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.259633 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a252f556-2d81-4164-8efa-65037946f382-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a252f556-2d81-4164-8efa-65037946f382-istiod-ca-cert\") pod \"a252f556-2d81-4164-8efa-65037946f382\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " Oct 30 19:41:02.259700 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.259664 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-istio-data\") pod \"1d06c47f-e741-4465-a0bb-63b6544002da\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " Oct 30 19:41:02.259969 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.259716 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d06c47f-e741-4465-a0bb-63b6544002da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d06c47f-e741-4465-a0bb-63b6544002da-istiod-ca-cert\") pod \"1d06c47f-e741-4465-a0bb-63b6544002da\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " Oct 30 19:41:02.259969 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.259869 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-credential-socket\") pod \"1d06c47f-e741-4465-a0bb-63b6544002da\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " Oct 30 19:41:02.259969 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.259908 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-workload-socket\") pod \"1d06c47f-e741-4465-a0bb-63b6544002da\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " Oct 30 19:41:02.260137 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.260054 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-workload-certs\") pod \"a252f556-2d81-4164-8efa-65037946f382\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " Oct 30 19:41:02.260137 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.260088 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-workload-certs\") pod \"1d06c47f-e741-4465-a0bb-63b6544002da\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " Oct 30 19:41:02.260137 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.260124 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d06c47f-e741-4465-a0bb-63b6544002da-istio-token\" (UniqueName: \"kubernetes.io/projected/1d06c47f-e741-4465-a0bb-63b6544002da-istio-token\") pod \"1d06c47f-e741-4465-a0bb-63b6544002da\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " Oct 30 19:41:02.260305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.260205 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-workload-socket\") pod \"a252f556-2d81-4164-8efa-65037946f382\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " Oct 30 19:41:02.260305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.260261 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-istio-envoy\") pod \"a252f556-2d81-4164-8efa-65037946f382\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " Oct 30 19:41:02.260305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.260289 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d06c47f-e741-4465-a0bb-63b6544002da-kube-api-access-5db5l\" (UniqueName: \"kubernetes.io/projected/1d06c47f-e741-4465-a0bb-63b6544002da-kube-api-access-5db5l\") pod \"1d06c47f-e741-4465-a0bb-63b6544002da\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " Oct 30 19:41:02.260466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.260318 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-istio-envoy\") pod \"1d06c47f-e741-4465-a0bb-63b6544002da\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " Oct 30 19:41:02.260466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.260355 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-credential-socket\") pod \"a252f556-2d81-4164-8efa-65037946f382\" (UID: \"a252f556-2d81-4164-8efa-65037946f382\") " Oct 30 19:41:02.260466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.260420 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d06c47f-e741-4465-a0bb-63b6544002da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d06c47f-e741-4465-a0bb-63b6544002da-istio-podinfo\") pod \"1d06c47f-e741-4465-a0bb-63b6544002da\" (UID: \"1d06c47f-e741-4465-a0bb-63b6544002da\") " Oct 30 19:41:02.271148 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.270993 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-workload-socket" pod "1d06c47f-e741-4465-a0bb-63b6544002da" (UID: "1d06c47f-e741-4465-a0bb-63b6544002da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:02.271436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.271403 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-istio-data" pod "a252f556-2d81-4164-8efa-65037946f382" (UID: "a252f556-2d81-4164-8efa-65037946f382"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:02.282863 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.273907 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a252f556-2d81-4164-8efa-65037946f382-istiod-ca-cert" pod "a252f556-2d81-4164-8efa-65037946f382" (UID: "a252f556-2d81-4164-8efa-65037946f382"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:02.282863 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.275221 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d06c47f-e741-4465-a0bb-63b6544002da-istiod-ca-cert" pod "1d06c47f-e741-4465-a0bb-63b6544002da" (UID: "1d06c47f-e741-4465-a0bb-63b6544002da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:02.282863 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.275491 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-istio-data" pod "1d06c47f-e741-4465-a0bb-63b6544002da" (UID: "1d06c47f-e741-4465-a0bb-63b6544002da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:02.282863 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.276630 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-workload-socket" pod "a252f556-2d81-4164-8efa-65037946f382" (UID: "a252f556-2d81-4164-8efa-65037946f382"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:02.282863 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.277107 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-workload-certs" pod "1d06c47f-e741-4465-a0bb-63b6544002da" (UID: "1d06c47f-e741-4465-a0bb-63b6544002da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:02.282863 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.279364 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-credential-socket" pod "1d06c47f-e741-4465-a0bb-63b6544002da" (UID: "1d06c47f-e741-4465-a0bb-63b6544002da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:02.282863 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.279421 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-workload-certs" pod "a252f556-2d81-4164-8efa-65037946f382" (UID: "a252f556-2d81-4164-8efa-65037946f382"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:02.309682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.294091 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:41:02.309682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.298782 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a252f556-2d81-4164-8efa-65037946f382-istio-token" pod "a252f556-2d81-4164-8efa-65037946f382" (UID: "a252f556-2d81-4164-8efa-65037946f382"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:02.309682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.305139 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d06c47f-e741-4465-a0bb-63b6544002da-kube-api-access-5db5l" pod "1d06c47f-e741-4465-a0bb-63b6544002da" (UID: "1d06c47f-e741-4465-a0bb-63b6544002da"). InnerVolumeSpecName "kube-api-access-5db5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:02.309682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.305793 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-istio-envoy" pod "a252f556-2d81-4164-8efa-65037946f382" (UID: "a252f556-2d81-4164-8efa-65037946f382"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:02.310480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.310448 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a252f556-2d81-4164-8efa-65037946f382-istio-podinfo" pod "a252f556-2d81-4164-8efa-65037946f382" (UID: "a252f556-2d81-4164-8efa-65037946f382"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:41:02.311113 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.305528 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-credential-socket" pod "a252f556-2d81-4164-8efa-65037946f382" (UID: "a252f556-2d81-4164-8efa-65037946f382"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:02.311938 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.311911 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a252f556-2d81-4164-8efa-65037946f382-kube-api-access-8tqng" pod "a252f556-2d81-4164-8efa-65037946f382" (UID: "a252f556-2d81-4164-8efa-65037946f382"). InnerVolumeSpecName "kube-api-access-8tqng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:02.317697 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.317510 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-istio-envoy" pod "1d06c47f-e741-4465-a0bb-63b6544002da" (UID: "1d06c47f-e741-4465-a0bb-63b6544002da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:02.317697 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.317550 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d06c47f-e741-4465-a0bb-63b6544002da-istio-token" pod "1d06c47f-e741-4465-a0bb-63b6544002da" (UID: "1d06c47f-e741-4465-a0bb-63b6544002da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:02.317697 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.317624 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d06c47f-e741-4465-a0bb-63b6544002da-istio-podinfo" pod "1d06c47f-e741-4465-a0bb-63b6544002da" (UID: "1d06c47f-e741-4465-a0bb-63b6544002da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:41:02.321047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.321017 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363315 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363351 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d06c47f-e741-4465-a0bb-63b6544002da-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363367 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363380 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tqng\" (UniqueName: \"kubernetes.io/projected/a252f556-2d81-4164-8efa-65037946f382-kube-api-access-8tqng\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363396 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a252f556-2d81-4164-8efa-65037946f382-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363408 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a252f556-2d81-4164-8efa-65037946f382-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363421 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a252f556-2d81-4164-8efa-65037946f382-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363434 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363447 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d06c47f-e741-4465-a0bb-63b6544002da-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363460 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363473 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363487 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363501 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363513 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d06c47f-e741-4465-a0bb-63b6544002da-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363525 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363537 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a252f556-2d81-4164-8efa-65037946f382-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363550 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5db5l\" (UniqueName: \"kubernetes.io/projected/1d06c47f-e741-4465-a0bb-63b6544002da-kube-api-access-5db5l\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.363857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.363563 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d06c47f-e741-4465-a0bb-63b6544002da-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.398717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.398679 5047 generic.go:356] "Generic (PLEG): container finished" podID="a252f556-2d81-4164-8efa-65037946f382" containerID="1283e93829ab77a30c4b384bddfd704c8bc6c269937b20cc979c006cb3bd82bd" exitCode=0 Oct 30 19:41:02.399311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.398929 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" event={"ID":"a252f556-2d81-4164-8efa-65037946f382","Type":"ContainerDied","Data":"1283e93829ab77a30c4b384bddfd704c8bc6c269937b20cc979c006cb3bd82bd"} Oct 30 19:41:02.401460 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.399502 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" event={"ID":"a252f556-2d81-4164-8efa-65037946f382","Type":"ContainerDied","Data":"9ddbe4e70debfc1f7faadc2e775c6617a49da7b10ae2d67277c0890cbde8fa69"} Oct 30 19:41:02.401460 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.399544 5047 scope.go:122] "RemoveContainer" containerID="1283e93829ab77a30c4b384bddfd704c8bc6c269937b20cc979c006cb3bd82bd" Oct 30 19:41:02.401460 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.399045 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs" Oct 30 19:41:02.408223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.403127 5047 generic.go:356] "Generic (PLEG): container finished" podID="908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" containerID="0d81bdb27dc6388fc7151f31d572e32217f14a8614b8f06b880b4a6ce622bf54" exitCode=0 Oct 30 19:41:02.408223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.403352 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" event={"ID":"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb","Type":"ContainerDied","Data":"0d81bdb27dc6388fc7151f31d572e32217f14a8614b8f06b880b4a6ce622bf54"} Oct 30 19:41:02.408223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.403403 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" event={"ID":"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb","Type":"ContainerDied","Data":"a6bdd32d42472e1eb1ec320cc76e2787af416b0114ca23d1f636878c819128e3"} Oct 30 19:41:02.408223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.403489 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk" Oct 30 19:41:02.408223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.405994 5047 generic.go:356] "Generic (PLEG): container finished" podID="1d06c47f-e741-4465-a0bb-63b6544002da" containerID="578dee23a51f24b7f45024b7c7b561f8f253d4fd9163925b4cb0943a40aab0e9" exitCode=0 Oct 30 19:41:02.408223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.406078 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" event={"ID":"1d06c47f-e741-4465-a0bb-63b6544002da","Type":"ContainerDied","Data":"578dee23a51f24b7f45024b7c7b561f8f253d4fd9163925b4cb0943a40aab0e9"} Oct 30 19:41:02.408223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.406130 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" event={"ID":"1d06c47f-e741-4465-a0bb-63b6544002da","Type":"ContainerDied","Data":"fc701028a455e4a6705f4a9ce57c25a7447be8b49502ef5ff5d4a6ac0a05832e"} Oct 30 19:41:02.408223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.406212 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp" Oct 30 19:41:02.439501 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.439409 5047 scope.go:122] "RemoveContainer" containerID="a1348b944313cd6a43577eb3d495ef73a1c9e16ed41a301461d8fa11773d4559" Oct 30 19:41:02.455798 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.455642 5047 scope.go:122] "RemoveContainer" containerID="1283e93829ab77a30c4b384bddfd704c8bc6c269937b20cc979c006cb3bd82bd" Oct 30 19:41:02.456079 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:02.456049 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1283e93829ab77a30c4b384bddfd704c8bc6c269937b20cc979c006cb3bd82bd\": container with ID starting with 1283e93829ab77a30c4b384bddfd704c8bc6c269937b20cc979c006cb3bd82bd not found: ID does not exist" containerID="1283e93829ab77a30c4b384bddfd704c8bc6c269937b20cc979c006cb3bd82bd" Oct 30 19:41:02.456144 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.456093 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1283e93829ab77a30c4b384bddfd704c8bc6c269937b20cc979c006cb3bd82bd"} err="failed to get container status \"1283e93829ab77a30c4b384bddfd704c8bc6c269937b20cc979c006cb3bd82bd\": rpc error: code = NotFound desc = could not find container \"1283e93829ab77a30c4b384bddfd704c8bc6c269937b20cc979c006cb3bd82bd\": container with ID starting with 1283e93829ab77a30c4b384bddfd704c8bc6c269937b20cc979c006cb3bd82bd not found: ID does not exist" Oct 30 19:41:02.456144 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.456119 5047 scope.go:122] "RemoveContainer" containerID="a1348b944313cd6a43577eb3d495ef73a1c9e16ed41a301461d8fa11773d4559" Oct 30 19:41:02.456584 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:02.456558 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1348b944313cd6a43577eb3d495ef73a1c9e16ed41a301461d8fa11773d4559\": container with ID starting with a1348b944313cd6a43577eb3d495ef73a1c9e16ed41a301461d8fa11773d4559 not found: ID does not exist" containerID="a1348b944313cd6a43577eb3d495ef73a1c9e16ed41a301461d8fa11773d4559" Oct 30 19:41:02.456680 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.456604 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1348b944313cd6a43577eb3d495ef73a1c9e16ed41a301461d8fa11773d4559"} err="failed to get container status \"a1348b944313cd6a43577eb3d495ef73a1c9e16ed41a301461d8fa11773d4559\": rpc error: code = NotFound desc = could not find container \"a1348b944313cd6a43577eb3d495ef73a1c9e16ed41a301461d8fa11773d4559\": container with ID starting with a1348b944313cd6a43577eb3d495ef73a1c9e16ed41a301461d8fa11773d4559 not found: ID does not exist" Oct 30 19:41:02.456680 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.456626 5047 scope.go:122] "RemoveContainer" containerID="0d81bdb27dc6388fc7151f31d572e32217f14a8614b8f06b880b4a6ce622bf54" Oct 30 19:41:02.465127 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.465085 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-envoy\") pod \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " Oct 30 19:41:02.465335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.465205 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istiod-ca-cert\") pod \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " Oct 30 19:41:02.465335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.465240 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-kube-api-access-wz59s\" (UniqueName: \"kubernetes.io/projected/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-kube-api-access-wz59s\") pod \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " Oct 30 19:41:02.465335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.465277 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-podinfo\") pod \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " Oct 30 19:41:02.465335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.465311 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-workload-socket\") pod \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " Oct 30 19:41:02.465577 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.465432 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-data\") pod \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " Oct 30 19:41:02.465641 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.465594 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-credential-socket\") pod \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " Oct 30 19:41:02.465714 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.465694 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-workload-certs\") pod \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " Oct 30 19:41:02.465793 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.465776 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-token\" (UniqueName: \"kubernetes.io/projected/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-token\") pod \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\" (UID: \"908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb\") " Oct 30 19:41:02.467123 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.467087 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istiod-ca-cert" pod "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" (UID: "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:02.467394 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.467303 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-credential-socket" pod "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" (UID: "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:02.467503 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.467478 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-data" pod "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" (UID: "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:02.467716 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.467688 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-workload-certs" pod "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" (UID: "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:02.468355 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.468327 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-workload-socket" pod "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" (UID: "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:02.468473 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.468449 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-envoy" pod "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" (UID: "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:02.470789 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.470744 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-kube-api-access-wz59s" pod "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" (UID: "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb"). InnerVolumeSpecName "kube-api-access-wz59s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:02.472533 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.472498 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-podinfo" pod "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" (UID: "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:41:02.473662 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.473639 5047 scope.go:122] "RemoveContainer" containerID="0d81bdb27dc6388fc7151f31d572e32217f14a8614b8f06b880b4a6ce622bf54" Oct 30 19:41:02.473770 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.473747 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-token" pod "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" (UID: "908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:02.474111 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:02.474083 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d81bdb27dc6388fc7151f31d572e32217f14a8614b8f06b880b4a6ce622bf54\": container with ID starting with 0d81bdb27dc6388fc7151f31d572e32217f14a8614b8f06b880b4a6ce622bf54 not found: ID does not exist" containerID="0d81bdb27dc6388fc7151f31d572e32217f14a8614b8f06b880b4a6ce622bf54" Oct 30 19:41:02.474208 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.474120 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d81bdb27dc6388fc7151f31d572e32217f14a8614b8f06b880b4a6ce622bf54"} err="failed to get container status \"0d81bdb27dc6388fc7151f31d572e32217f14a8614b8f06b880b4a6ce622bf54\": rpc error: code = NotFound desc = could not find container \"0d81bdb27dc6388fc7151f31d572e32217f14a8614b8f06b880b4a6ce622bf54\": container with ID starting with 0d81bdb27dc6388fc7151f31d572e32217f14a8614b8f06b880b4a6ce622bf54 not found: ID does not exist" Oct 30 19:41:02.474208 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.474147 5047 scope.go:122] "RemoveContainer" containerID="578dee23a51f24b7f45024b7c7b561f8f253d4fd9163925b4cb0943a40aab0e9" Oct 30 19:41:02.509982 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.491892 5047 scope.go:122] "RemoveContainer" containerID="578dee23a51f24b7f45024b7c7b561f8f253d4fd9163925b4cb0943a40aab0e9" Oct 30 19:41:02.509982 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:02.493808 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578dee23a51f24b7f45024b7c7b561f8f253d4fd9163925b4cb0943a40aab0e9\": container with ID starting with 578dee23a51f24b7f45024b7c7b561f8f253d4fd9163925b4cb0943a40aab0e9 not found: ID does not exist" containerID="578dee23a51f24b7f45024b7c7b561f8f253d4fd9163925b4cb0943a40aab0e9" Oct 30 19:41:02.509982 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.493852 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578dee23a51f24b7f45024b7c7b561f8f253d4fd9163925b4cb0943a40aab0e9"} err="failed to get container status \"578dee23a51f24b7f45024b7c7b561f8f253d4fd9163925b4cb0943a40aab0e9\": rpc error: code = NotFound desc = could not find container \"578dee23a51f24b7f45024b7c7b561f8f253d4fd9163925b4cb0943a40aab0e9\": container with ID starting with 578dee23a51f24b7f45024b7c7b561f8f253d4fd9163925b4cb0943a40aab0e9 not found: ID does not exist" Oct 30 19:41:02.524085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.515909 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs"] Oct 30 19:41:02.524085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.523457 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-xqcxs"] Oct 30 19:41:02.531778 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.531746 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp"] Oct 30 19:41:02.538699 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.538664 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-7cksp"] Oct 30 19:41:02.569194 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.569151 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.569194 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.569189 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.569407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.569206 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.569407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.569219 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.569407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.569231 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.569407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.569244 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz59s\" (UniqueName: \"kubernetes.io/projected/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-kube-api-access-wz59s\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.569407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.569271 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.569407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.569285 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.569407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.569300 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:02.751523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.751487 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk"] Oct 30 19:41:02.760329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.760291 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-lvcqk"] Oct 30 19:41:02.889223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.889174 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf"] Oct 30 19:41:02.890545 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.890512 5047 cpu_manager.go:414] "containerMap: removing container" podUID="a252f556-2d81-4164-8efa-65037946f382" containerName="istio-proxy" Oct 30 19:41:02.890545 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.890542 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a252f556-2d81-4164-8efa-65037946f382" containerName="istio-proxy" Oct 30 19:41:02.890733 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.890583 5047 cpu_manager.go:414] "containerMap: removing container" podUID="d4d691a1-d91c-4985-9866-3a96f94b2e67" containerName="maas-api" Oct 30 19:41:02.890733 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.890592 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d691a1-d91c-4985-9866-3a96f94b2e67" containerName="maas-api" Oct 30 19:41:02.890733 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.890621 5047 cpu_manager.go:414] "containerMap: removing container" podUID="1d06c47f-e741-4465-a0bb-63b6544002da" containerName="istio-proxy" Oct 30 19:41:02.890733 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.890629 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d06c47f-e741-4465-a0bb-63b6544002da" containerName="istio-proxy" Oct 30 19:41:02.890733 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.890696 5047 cpu_manager.go:414] "containerMap: removing container" podUID="908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" containerName="istio-proxy" Oct 30 19:41:02.890733 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.890704 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" containerName="istio-proxy" Oct 30 19:41:02.890928 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.890915 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="a252f556-2d81-4164-8efa-65037946f382" containerName="istio-proxy" Oct 30 19:41:02.890965 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.890932 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d691a1-d91c-4985-9866-3a96f94b2e67" containerName="maas-api" Oct 30 19:41:02.890965 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.890945 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d06c47f-e741-4465-a0bb-63b6544002da" containerName="istio-proxy" Oct 30 19:41:02.890965 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.890959 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" containerName="istio-proxy" Oct 30 19:41:02.896306 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.896278 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:02.915393 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.913713 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-f44f74-openshift-default-dockercfg-jkbtg\"" Oct 30 19:41:02.925043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.922207 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf"] Oct 30 19:41:02.976998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.976871 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-workload-certs\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:02.976998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.976953 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hlgn\" (UniqueName: \"kubernetes.io/projected/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-kube-api-access-5hlgn\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:02.977285 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.977136 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-podinfo\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:02.977285 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.977251 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-data\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:02.977419 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.977399 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-envoy\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:02.977476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.977433 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istiod-ca-cert\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:02.977476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.977458 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-credential-socket\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:02.977594 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.977547 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-token\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:02.977667 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:02.977651 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-workload-socket\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.053187 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.053155 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d06c47f-e741-4465-a0bb-63b6544002da" path="/var/lib/kubelet/pods/1d06c47f-e741-4465-a0bb-63b6544002da/volumes" Oct 30 19:41:03.053799 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.053784 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb" path="/var/lib/kubelet/pods/908307f7-c52b-4bdc-b5f9-5d7ea7ea57eb/volumes" Oct 30 19:41:03.054377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.054361 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a252f556-2d81-4164-8efa-65037946f382" path="/var/lib/kubelet/pods/a252f556-2d81-4164-8efa-65037946f382/volumes" Oct 30 19:41:03.079444 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.079404 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-workload-certs\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.079718 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.079454 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlgn\" (UniqueName: \"kubernetes.io/projected/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-kube-api-access-5hlgn\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.079718 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.079500 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-podinfo\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.079718 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.079544 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-data\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.079718 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.079681 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-envoy\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.079718 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.079713 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istiod-ca-cert\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.079996 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.079745 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-credential-socket\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.079996 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.079789 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-token\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.079996 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.079910 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-workload-socket\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.079996 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.079932 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-workload-certs\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.079996 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.080028 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-data\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.080313 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.080211 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-credential-socket\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.080395 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.080372 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-workload-socket\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.080484 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.080466 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istiod-ca-cert\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.082171 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.082143 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-envoy\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.082501 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.082469 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-podinfo\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.107355 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.100709 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-token\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.107355 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.101144 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlgn\" (UniqueName: \"kubernetes.io/projected/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-kube-api-access-5hlgn\") pod \"e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.231100 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.230185 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:03.740897 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:41:03.740860 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec5c9c7_1994_4064_b2fe_ef9f288c53ba.slice/crio-19fff0724565a3ad340924257bff05ebe5084c18c81a59ba0f31a8cf374c02bc WatchSource:0}: Error finding container 19fff0724565a3ad340924257bff05ebe5084c18c81a59ba0f31a8cf374c02bc: Status 404 returned error can't find the container with id 19fff0724565a3ad340924257bff05ebe5084c18c81a59ba0f31a8cf374c02bc Oct 30 19:41:03.741193 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.741168 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf"] Oct 30 19:41:03.744526 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.744481 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:41:03.744651 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.744572 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:41:03.744651 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:03.744617 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:41:04.478592 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:04.478554 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" event={"ID":"dec5c9c7-1994-4064-b2fe-ef9f288c53ba","Type":"ContainerStarted","Data":"fd3e459e68032a1e32b941284916d7d31a89b7d1df2d912944e62cf9201d5d6e"} Oct 30 19:41:04.478983 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:04.478613 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" event={"ID":"dec5c9c7-1994-4064-b2fe-ef9f288c53ba","Type":"ContainerStarted","Data":"19fff0724565a3ad340924257bff05ebe5084c18c81a59ba0f31a8cf374c02bc"} Oct 30 19:41:04.538229 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:04.538168 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" podStartSLOduration=2.5381513350000002 podStartE2EDuration="2.538151335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:04.524375325 +0000 UTC m=+2217.598830684" watchObservedRunningTime="2026-10-30 19:41:04.538151335 +0000 UTC m=+2217.612606770" Oct 30 19:41:05.230711 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:05.230677 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:05.237166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:05.237130 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:05.487545 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:05.487441 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:05.488562 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:05.488540 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:41:07.182030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:07.181326 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf"] Oct 30 19:41:07.775539 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:07.775489 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9"] Oct 30 19:41:07.776035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:07.775990 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" podUID="1713eb56-d041-40ae-824a-3bdc1ca843f9" containerName="istio-proxy" containerID="cri-o://e631a021dd9bb5db2053ea0f3b362a219e96ea09c0532318483bbf4dc8be2baa" gracePeriod=30 Oct 30 19:41:09.605967 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:09.605928 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:41:10.179990 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:10.179955 5047 scope.go:122] "RemoveContainer" containerID="d7186411c8b531586afb6bef871723146d1f7042d10103c8f08c1bcac37e42fe" Oct 30 19:41:11.142202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:11.142156 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" podUID="6a0824a6-857e-47a0-9bff-5b0ed8ce38e9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:41:12.164782 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:12.164722 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" podUID="7d3adfad-8cca-4e98-9812-8c17c9cfb4c1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:41:13.080741 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.080709 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:41:13.190299 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.190264 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-workload-socket\") pod \"1713eb56-d041-40ae-824a-3bdc1ca843f9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " Oct 30 19:41:13.190761 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.190392 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-credential-socket\") pod \"1713eb56-d041-40ae-824a-3bdc1ca843f9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " Oct 30 19:41:13.190761 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.190488 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1713eb56-d041-40ae-824a-3bdc1ca843f9-kube-api-access-hmmsp\" (UniqueName: \"kubernetes.io/projected/1713eb56-d041-40ae-824a-3bdc1ca843f9-kube-api-access-hmmsp\") pod \"1713eb56-d041-40ae-824a-3bdc1ca843f9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " Oct 30 19:41:13.190761 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.190523 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-token\" (UniqueName: \"kubernetes.io/projected/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-token\") pod \"1713eb56-d041-40ae-824a-3bdc1ca843f9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " Oct 30 19:41:13.190761 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.190654 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-podinfo\") pod \"1713eb56-d041-40ae-824a-3bdc1ca843f9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " Oct 30 19:41:13.190761 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.190650 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-workload-socket" pod "1713eb56-d041-40ae-824a-3bdc1ca843f9" (UID: "1713eb56-d041-40ae-824a-3bdc1ca843f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:13.190761 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.190711 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-workload-certs\") pod \"1713eb56-d041-40ae-824a-3bdc1ca843f9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " Oct 30 19:41:13.191124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.190767 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1713eb56-d041-40ae-824a-3bdc1ca843f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1713eb56-d041-40ae-824a-3bdc1ca843f9-istiod-ca-cert\") pod \"1713eb56-d041-40ae-824a-3bdc1ca843f9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " Oct 30 19:41:13.191124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.190886 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-credential-socket" pod "1713eb56-d041-40ae-824a-3bdc1ca843f9" (UID: "1713eb56-d041-40ae-824a-3bdc1ca843f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:13.191124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.190911 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-data\") pod \"1713eb56-d041-40ae-824a-3bdc1ca843f9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " Oct 30 19:41:13.191124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.190959 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-envoy\") pod \"1713eb56-d041-40ae-824a-3bdc1ca843f9\" (UID: \"1713eb56-d041-40ae-824a-3bdc1ca843f9\") " Oct 30 19:41:13.191328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.191145 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-workload-certs" pod "1713eb56-d041-40ae-824a-3bdc1ca843f9" (UID: "1713eb56-d041-40ae-824a-3bdc1ca843f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:13.192206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.191601 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.192206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.191629 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.192206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.191646 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.192206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.191722 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1713eb56-d041-40ae-824a-3bdc1ca843f9-istiod-ca-cert" pod "1713eb56-d041-40ae-824a-3bdc1ca843f9" (UID: "1713eb56-d041-40ae-824a-3bdc1ca843f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:13.192206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.192152 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-data" pod "1713eb56-d041-40ae-824a-3bdc1ca843f9" (UID: "1713eb56-d041-40ae-824a-3bdc1ca843f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:13.193460 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.193430 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1713eb56-d041-40ae-824a-3bdc1ca843f9-kube-api-access-hmmsp" pod "1713eb56-d041-40ae-824a-3bdc1ca843f9" (UID: "1713eb56-d041-40ae-824a-3bdc1ca843f9"). InnerVolumeSpecName "kube-api-access-hmmsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:13.194082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.194042 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-podinfo" pod "1713eb56-d041-40ae-824a-3bdc1ca843f9" (UID: "1713eb56-d041-40ae-824a-3bdc1ca843f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:41:13.194315 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.194155 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-token" pod "1713eb56-d041-40ae-824a-3bdc1ca843f9" (UID: "1713eb56-d041-40ae-824a-3bdc1ca843f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:13.205063 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.201299 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-envoy" pod "1713eb56-d041-40ae-824a-3bdc1ca843f9" (UID: "1713eb56-d041-40ae-824a-3bdc1ca843f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:13.292985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.292881 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1713eb56-d041-40ae-824a-3bdc1ca843f9-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.292985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.292922 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.292985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.292935 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.292985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.292947 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmmsp\" (UniqueName: \"kubernetes.io/projected/1713eb56-d041-40ae-824a-3bdc1ca843f9-kube-api-access-hmmsp\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.292985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.292965 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.292985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.292978 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1713eb56-d041-40ae-824a-3bdc1ca843f9-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:13.606607 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.606567 5047 generic.go:356] "Generic (PLEG): container finished" podID="1713eb56-d041-40ae-824a-3bdc1ca843f9" containerID="e631a021dd9bb5db2053ea0f3b362a219e96ea09c0532318483bbf4dc8be2baa" exitCode=0 Oct 30 19:41:13.606832 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.606634 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" Oct 30 19:41:13.606832 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.606630 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" event={"ID":"1713eb56-d041-40ae-824a-3bdc1ca843f9","Type":"ContainerDied","Data":"e631a021dd9bb5db2053ea0f3b362a219e96ea09c0532318483bbf4dc8be2baa"} Oct 30 19:41:13.606832 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.606703 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9" event={"ID":"1713eb56-d041-40ae-824a-3bdc1ca843f9","Type":"ContainerDied","Data":"a527ab04574f6b5c3d20467878569258fbc0916c670c8faa6f618201f5ff21ba"} Oct 30 19:41:13.606832 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.606734 5047 scope.go:122] "RemoveContainer" containerID="e631a021dd9bb5db2053ea0f3b362a219e96ea09c0532318483bbf4dc8be2baa" Oct 30 19:41:13.620122 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.620091 5047 scope.go:122] "RemoveContainer" containerID="501f90f756a7578030e096d0b1936d121121055b50a97aff3e43ae496d9a223f" Oct 30 19:41:13.631257 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.631226 5047 scope.go:122] "RemoveContainer" containerID="e631a021dd9bb5db2053ea0f3b362a219e96ea09c0532318483bbf4dc8be2baa" Oct 30 19:41:13.631570 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:13.631548 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e631a021dd9bb5db2053ea0f3b362a219e96ea09c0532318483bbf4dc8be2baa\": container with ID starting with e631a021dd9bb5db2053ea0f3b362a219e96ea09c0532318483bbf4dc8be2baa not found: ID does not exist" containerID="e631a021dd9bb5db2053ea0f3b362a219e96ea09c0532318483bbf4dc8be2baa" Oct 30 19:41:13.631648 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.631583 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e631a021dd9bb5db2053ea0f3b362a219e96ea09c0532318483bbf4dc8be2baa"} err="failed to get container status \"e631a021dd9bb5db2053ea0f3b362a219e96ea09c0532318483bbf4dc8be2baa\": rpc error: code = NotFound desc = could not find container \"e631a021dd9bb5db2053ea0f3b362a219e96ea09c0532318483bbf4dc8be2baa\": container with ID starting with e631a021dd9bb5db2053ea0f3b362a219e96ea09c0532318483bbf4dc8be2baa not found: ID does not exist" Oct 30 19:41:13.631648 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.631609 5047 scope.go:122] "RemoveContainer" containerID="501f90f756a7578030e096d0b1936d121121055b50a97aff3e43ae496d9a223f" Oct 30 19:41:13.631930 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:13.631910 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"501f90f756a7578030e096d0b1936d121121055b50a97aff3e43ae496d9a223f\": container with ID starting with 501f90f756a7578030e096d0b1936d121121055b50a97aff3e43ae496d9a223f not found: ID does not exist" containerID="501f90f756a7578030e096d0b1936d121121055b50a97aff3e43ae496d9a223f" Oct 30 19:41:13.632000 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.631942 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"501f90f756a7578030e096d0b1936d121121055b50a97aff3e43ae496d9a223f"} err="failed to get container status \"501f90f756a7578030e096d0b1936d121121055b50a97aff3e43ae496d9a223f\": rpc error: code = NotFound desc = could not find container \"501f90f756a7578030e096d0b1936d121121055b50a97aff3e43ae496d9a223f\": container with ID starting with 501f90f756a7578030e096d0b1936d121121055b50a97aff3e43ae496d9a223f not found: ID does not exist" Oct 30 19:41:13.654393 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.654350 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9"] Oct 30 19:41:13.663804 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:13.663772 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f4ab4a-openshift-default-688bb6f67f-646k9"] Oct 30 19:41:15.052024 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:15.051944 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1713eb56-d041-40ae-824a-3bdc1ca843f9" path="/var/lib/kubelet/pods/1713eb56-d041-40ae-824a-3bdc1ca843f9/volumes" Oct 30 19:41:16.101981 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.101950 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" Oct 30 19:41:16.149182 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.149142 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d3adfad-8cca-4e98-9812-8c17c9cfb4c1-kube-api-access-g6c9t\" (UniqueName: \"kubernetes.io/projected/7d3adfad-8cca-4e98-9812-8c17c9cfb4c1-kube-api-access-g6c9t\") pod \"7d3adfad-8cca-4e98-9812-8c17c9cfb4c1\" (UID: \"7d3adfad-8cca-4e98-9812-8c17c9cfb4c1\") " Oct 30 19:41:16.149495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.149477 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" Oct 30 19:41:16.149610 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.149480 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d3adfad-8cca-4e98-9812-8c17c9cfb4c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d3adfad-8cca-4e98-9812-8c17c9cfb4c1-plugins-config-volume\") pod \"7d3adfad-8cca-4e98-9812-8c17c9cfb4c1\" (UID: \"7d3adfad-8cca-4e98-9812-8c17c9cfb4c1\") " Oct 30 19:41:16.149863 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.149837 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d3adfad-8cca-4e98-9812-8c17c9cfb4c1-plugins-config-volume" pod "7d3adfad-8cca-4e98-9812-8c17c9cfb4c1" (UID: "7d3adfad-8cca-4e98-9812-8c17c9cfb4c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:16.150309 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.150290 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d3adfad-8cca-4e98-9812-8c17c9cfb4c1-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:16.151331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.151312 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d3adfad-8cca-4e98-9812-8c17c9cfb4c1-kube-api-access-g6c9t" pod "7d3adfad-8cca-4e98-9812-8c17c9cfb4c1" (UID: "7d3adfad-8cca-4e98-9812-8c17c9cfb4c1"). InnerVolumeSpecName "kube-api-access-g6c9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:16.251683 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.251637 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0824a6-857e-47a0-9bff-5b0ed8ce38e9-kube-api-access-ht2c7\" (UniqueName: \"kubernetes.io/projected/6a0824a6-857e-47a0-9bff-5b0ed8ce38e9-kube-api-access-ht2c7\") pod \"6a0824a6-857e-47a0-9bff-5b0ed8ce38e9\" (UID: \"6a0824a6-857e-47a0-9bff-5b0ed8ce38e9\") " Oct 30 19:41:16.251931 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.251912 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a0824a6-857e-47a0-9bff-5b0ed8ce38e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0824a6-857e-47a0-9bff-5b0ed8ce38e9-plugins-config-volume\") pod \"6a0824a6-857e-47a0-9bff-5b0ed8ce38e9\" (UID: \"6a0824a6-857e-47a0-9bff-5b0ed8ce38e9\") " Oct 30 19:41:16.252348 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.252319 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0824a6-857e-47a0-9bff-5b0ed8ce38e9-plugins-config-volume" pod "6a0824a6-857e-47a0-9bff-5b0ed8ce38e9" (UID: "6a0824a6-857e-47a0-9bff-5b0ed8ce38e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:16.252672 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.252656 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0824a6-857e-47a0-9bff-5b0ed8ce38e9-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:16.252718 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.252675 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6c9t\" (UniqueName: \"kubernetes.io/projected/7d3adfad-8cca-4e98-9812-8c17c9cfb4c1-kube-api-access-g6c9t\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:16.253878 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.253858 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0824a6-857e-47a0-9bff-5b0ed8ce38e9-kube-api-access-ht2c7" pod "6a0824a6-857e-47a0-9bff-5b0ed8ce38e9" (UID: "6a0824a6-857e-47a0-9bff-5b0ed8ce38e9"). InnerVolumeSpecName "kube-api-access-ht2c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:16.353947 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.353913 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht2c7\" (UniqueName: \"kubernetes.io/projected/6a0824a6-857e-47a0-9bff-5b0ed8ce38e9-kube-api-access-ht2c7\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:16.632360 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.632272 5047 generic.go:356] "Generic (PLEG): container finished" podID="7d3adfad-8cca-4e98-9812-8c17c9cfb4c1" containerID="c7e2592193c532f309801930c9dea7af8868a96f3f98a21263c029c1b579251a" exitCode=0 Oct 30 19:41:16.632360 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.632316 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" Oct 30 19:41:16.632360 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.632320 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" event={"ID":"7d3adfad-8cca-4e98-9812-8c17c9cfb4c1","Type":"ContainerDied","Data":"c7e2592193c532f309801930c9dea7af8868a96f3f98a21263c029c1b579251a"} Oct 30 19:41:16.632625 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.632380 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x" event={"ID":"7d3adfad-8cca-4e98-9812-8c17c9cfb4c1","Type":"ContainerDied","Data":"baae557c64746b2fed340fe42e5bf1a361ca89352ce65c828d00d87aad1aaf93"} Oct 30 19:41:16.632625 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.632405 5047 scope.go:122] "RemoveContainer" containerID="c7e2592193c532f309801930c9dea7af8868a96f3f98a21263c029c1b579251a" Oct 30 19:41:16.634193 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.634173 5047 generic.go:356] "Generic (PLEG): container finished" podID="6a0824a6-857e-47a0-9bff-5b0ed8ce38e9" containerID="f49f3c9ec78854bc03d585c8a1255199dbe3b8e9e51e99213afe8e9ba2df949c" exitCode=0 Oct 30 19:41:16.634323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.634210 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" event={"ID":"6a0824a6-857e-47a0-9bff-5b0ed8ce38e9","Type":"ContainerDied","Data":"f49f3c9ec78854bc03d585c8a1255199dbe3b8e9e51e99213afe8e9ba2df949c"} Oct 30 19:41:16.634323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.634245 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" Oct 30 19:41:16.634425 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.634247 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l" event={"ID":"6a0824a6-857e-47a0-9bff-5b0ed8ce38e9","Type":"ContainerDied","Data":"da682d3050e5f4579803c676304e7bd9534898e3a0eeffd4e2354cc726274124"} Oct 30 19:41:16.645268 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.644856 5047 scope.go:122] "RemoveContainer" containerID="c7e2592193c532f309801930c9dea7af8868a96f3f98a21263c029c1b579251a" Oct 30 19:41:16.646693 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:16.646651 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e2592193c532f309801930c9dea7af8868a96f3f98a21263c029c1b579251a\": container with ID starting with c7e2592193c532f309801930c9dea7af8868a96f3f98a21263c029c1b579251a not found: ID does not exist" containerID="c7e2592193c532f309801930c9dea7af8868a96f3f98a21263c029c1b579251a" Oct 30 19:41:16.646911 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.646883 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e2592193c532f309801930c9dea7af8868a96f3f98a21263c029c1b579251a"} err="failed to get container status \"c7e2592193c532f309801930c9dea7af8868a96f3f98a21263c029c1b579251a\": rpc error: code = NotFound desc = could not find container \"c7e2592193c532f309801930c9dea7af8868a96f3f98a21263c029c1b579251a\": container with ID starting with c7e2592193c532f309801930c9dea7af8868a96f3f98a21263c029c1b579251a not found: ID does not exist" Oct 30 19:41:16.647050 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.647033 5047 scope.go:122] "RemoveContainer" containerID="f49f3c9ec78854bc03d585c8a1255199dbe3b8e9e51e99213afe8e9ba2df949c" Oct 30 19:41:16.668862 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.668833 5047 scope.go:122] "RemoveContainer" containerID="f49f3c9ec78854bc03d585c8a1255199dbe3b8e9e51e99213afe8e9ba2df949c" Oct 30 19:41:16.669285 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:16.669260 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49f3c9ec78854bc03d585c8a1255199dbe3b8e9e51e99213afe8e9ba2df949c\": container with ID starting with f49f3c9ec78854bc03d585c8a1255199dbe3b8e9e51e99213afe8e9ba2df949c not found: ID does not exist" containerID="f49f3c9ec78854bc03d585c8a1255199dbe3b8e9e51e99213afe8e9ba2df949c" Oct 30 19:41:16.669371 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.669295 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49f3c9ec78854bc03d585c8a1255199dbe3b8e9e51e99213afe8e9ba2df949c"} err="failed to get container status \"f49f3c9ec78854bc03d585c8a1255199dbe3b8e9e51e99213afe8e9ba2df949c\": rpc error: code = NotFound desc = could not find container \"f49f3c9ec78854bc03d585c8a1255199dbe3b8e9e51e99213afe8e9ba2df949c\": container with ID starting with f49f3c9ec78854bc03d585c8a1255199dbe3b8e9e51e99213afe8e9ba2df949c not found: ID does not exist" Oct 30 19:41:16.692251 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.692213 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l"] Oct 30 19:41:16.698946 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.698911 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4ab4a-67d7c4596c-q7j5l"] Oct 30 19:41:16.727803 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.727763 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x"] Oct 30 19:41:16.733739 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:16.733689 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4ab4a-58874dbb4d-fsv9x"] Oct 30 19:41:17.052932 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:17.052849 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0824a6-857e-47a0-9bff-5b0ed8ce38e9" path="/var/lib/kubelet/pods/6a0824a6-857e-47a0-9bff-5b0ed8ce38e9/volumes" Oct 30 19:41:17.053530 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:17.053509 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d3adfad-8cca-4e98-9812-8c17c9cfb4c1" path="/var/lib/kubelet/pods/7d3adfad-8cca-4e98-9812-8c17c9cfb4c1/volumes" Oct 30 19:41:18.760787 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.760744 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9"] Oct 30 19:41:18.761569 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.761550 5047 cpu_manager.go:414] "containerMap: removing container" podUID="7d3adfad-8cca-4e98-9812-8c17c9cfb4c1" containerName="payload-processing" Oct 30 19:41:18.761616 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.761570 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d3adfad-8cca-4e98-9812-8c17c9cfb4c1" containerName="payload-processing" Oct 30 19:41:18.761616 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.761597 5047 cpu_manager.go:414] "containerMap: removing container" podUID="a252f556-2d81-4164-8efa-65037946f382" containerName="istio-proxy" Oct 30 19:41:18.761616 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.761603 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a252f556-2d81-4164-8efa-65037946f382" containerName="istio-proxy" Oct 30 19:41:18.761723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.761616 5047 cpu_manager.go:414] "containerMap: removing container" podUID="6a0824a6-857e-47a0-9bff-5b0ed8ce38e9" containerName="payload-pre-processing" Oct 30 19:41:18.761723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.761622 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0824a6-857e-47a0-9bff-5b0ed8ce38e9" containerName="payload-pre-processing" Oct 30 19:41:18.761723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.761629 5047 cpu_manager.go:414] "containerMap: removing container" podUID="1713eb56-d041-40ae-824a-3bdc1ca843f9" containerName="istio-proxy" Oct 30 19:41:18.761723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.761634 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1713eb56-d041-40ae-824a-3bdc1ca843f9" containerName="istio-proxy" Oct 30 19:41:18.761723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.761643 5047 cpu_manager.go:414] "containerMap: removing container" podUID="1713eb56-d041-40ae-824a-3bdc1ca843f9" containerName="istio-proxy" Oct 30 19:41:18.761723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.761648 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1713eb56-d041-40ae-824a-3bdc1ca843f9" containerName="istio-proxy" Oct 30 19:41:18.761901 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.761777 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="a252f556-2d81-4164-8efa-65037946f382" containerName="istio-proxy" Oct 30 19:41:18.761901 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.761787 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0824a6-857e-47a0-9bff-5b0ed8ce38e9" containerName="payload-pre-processing" Oct 30 19:41:18.761901 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.761795 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="1713eb56-d041-40ae-824a-3bdc1ca843f9" containerName="istio-proxy" Oct 30 19:41:18.761901 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.761801 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="1713eb56-d041-40ae-824a-3bdc1ca843f9" containerName="istio-proxy" Oct 30 19:41:18.761901 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.761815 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d3adfad-8cca-4e98-9812-8c17c9cfb4c1" containerName="payload-processing" Oct 30 19:41:18.767282 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.767259 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" Oct 30 19:41:18.782771 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.782715 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-f44f74\"" Oct 30 19:41:18.789138 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.789101 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9"] Oct 30 19:41:18.791654 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.791630 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-f44f74-dockercfg-k77g4\"" Oct 30 19:41:18.797768 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.797738 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w"] Oct 30 19:41:18.802487 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.802461 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" Oct 30 19:41:18.823880 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.823842 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w"] Oct 30 19:41:19.003030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:18.996241 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfw6n\" (UniqueName: \"kubernetes.io/projected/d3d82371-f264-41cc-b5d1-5cb5f5b35086-kube-api-access-vfw6n\") pod \"payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9\" (UID: \"d3d82371-f264-41cc-b5d1-5cb5f5b35086\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" Oct 30 19:41:19.036212 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:19.036112 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d82371-f264-41cc-b5d1-5cb5f5b35086-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9\" (UID: \"d3d82371-f264-41cc-b5d1-5cb5f5b35086\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" Oct 30 19:41:19.070823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:19.070397 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-249sj\" (UniqueName: \"kubernetes.io/projected/6c9f0447-ecc0-4d98-95fe-3d11df134549-kube-api-access-249sj\") pod \"payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w\" (UID: \"6c9f0447-ecc0-4d98-95fe-3d11df134549\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" Oct 30 19:41:19.071538 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:19.071510 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9f0447-ecc0-4d98-95fe-3d11df134549-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w\" (UID: \"6c9f0447-ecc0-4d98-95fe-3d11df134549\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" Oct 30 19:41:19.194735 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:19.194690 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d82371-f264-41cc-b5d1-5cb5f5b35086-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9\" (UID: \"d3d82371-f264-41cc-b5d1-5cb5f5b35086\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" Oct 30 19:41:19.197208 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:19.197177 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-249sj\" (UniqueName: \"kubernetes.io/projected/6c9f0447-ecc0-4d98-95fe-3d11df134549-kube-api-access-249sj\") pod \"payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w\" (UID: \"6c9f0447-ecc0-4d98-95fe-3d11df134549\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" Oct 30 19:41:19.197820 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:19.197795 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9f0447-ecc0-4d98-95fe-3d11df134549-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w\" (UID: \"6c9f0447-ecc0-4d98-95fe-3d11df134549\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" Oct 30 19:41:19.198922 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:19.196262 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d82371-f264-41cc-b5d1-5cb5f5b35086-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9\" (UID: \"d3d82371-f264-41cc-b5d1-5cb5f5b35086\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" Oct 30 19:41:19.199100 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:19.198932 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfw6n\" (UniqueName: \"kubernetes.io/projected/d3d82371-f264-41cc-b5d1-5cb5f5b35086-kube-api-access-vfw6n\") pod \"payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9\" (UID: \"d3d82371-f264-41cc-b5d1-5cb5f5b35086\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" Oct 30 19:41:19.199100 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:19.198788 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9f0447-ecc0-4d98-95fe-3d11df134549-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w\" (UID: \"6c9f0447-ecc0-4d98-95fe-3d11df134549\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" Oct 30 19:41:19.215436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:19.215399 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-249sj\" (UniqueName: \"kubernetes.io/projected/6c9f0447-ecc0-4d98-95fe-3d11df134549-kube-api-access-249sj\") pod \"payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w\" (UID: \"6c9f0447-ecc0-4d98-95fe-3d11df134549\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" Oct 30 19:41:19.216147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:19.216124 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfw6n\" (UniqueName: \"kubernetes.io/projected/d3d82371-f264-41cc-b5d1-5cb5f5b35086-kube-api-access-vfw6n\") pod \"payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9\" (UID: \"d3d82371-f264-41cc-b5d1-5cb5f5b35086\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" Oct 30 19:41:19.383026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:19.382971 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" Oct 30 19:41:19.420101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:19.420060 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" Oct 30 19:41:19.729962 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:19.729932 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9"] Oct 30 19:41:19.773114 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:19.773083 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w"] Oct 30 19:41:20.763615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:20.763570 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" event={"ID":"d3d82371-f264-41cc-b5d1-5cb5f5b35086","Type":"ContainerStarted","Data":"fbb17375dbca5989e7bdd73409408930492eb8c576daa129df6a2bb86abab1f1"} Oct 30 19:41:20.772512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:20.772483 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" Oct 30 19:41:20.772813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:20.772798 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" Oct 30 19:41:20.772942 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:20.772924 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" event={"ID":"d3d82371-f264-41cc-b5d1-5cb5f5b35086","Type":"ContainerStarted","Data":"0933d5df9e19ac393cf686114aa2c1403932ae46b8aebfab1985ee668a55cab9"} Oct 30 19:41:20.773088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:20.773073 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" event={"ID":"6c9f0447-ecc0-4d98-95fe-3d11df134549","Type":"ContainerStarted","Data":"7afc270683388ea1a265556bfa460f56d0ff3e52042e6ac16171018246a225ba"} Oct 30 19:41:20.773202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:20.773183 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" event={"ID":"6c9f0447-ecc0-4d98-95fe-3d11df134549","Type":"ContainerStarted","Data":"ef92f274ca99a571e3ce998145ad99dd7ca518e250a880c85fe5529f94f59529"} Oct 30 19:41:20.885039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:20.861388 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" podStartSLOduration=2.86136451 podStartE2EDuration="2.86136451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:20.848693723 +0000 UTC m=+2233.923149083" watchObservedRunningTime="2026-10-30 19:41:20.86136451 +0000 UTC m=+2233.935819874" Oct 30 19:41:20.911625 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:20.911557 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" podStartSLOduration=2.911533972 podStartE2EDuration="2.911533972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:20.907270825 +0000 UTC m=+2233.981726186" watchObservedRunningTime="2026-10-30 19:41:20.911533972 +0000 UTC m=+2233.985989332" Oct 30 19:41:22.221028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:22.218227 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" podUID="7456b22a-9c80-4fbb-9236-08eed4067902" containerName="main" containerID="cri-o://c8ad6e743a0419c65ea38f517284ca2c155a5070c5ab8d0c8824502908afc6de" gracePeriod=45 Oct 30 19:41:22.239069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:22.230244 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" podUID="7456b22a-9c80-4fbb-9236-08eed4067902" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": read tcp 10.132.0.2:35464->10.132.0.143:8000: read: connection reset by peer" Oct 30 19:41:22.841948 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:22.841910 5047 generic.go:356] "Generic (PLEG): container finished" podID="7456b22a-9c80-4fbb-9236-08eed4067902" containerID="c8ad6e743a0419c65ea38f517284ca2c155a5070c5ab8d0c8824502908afc6de" exitCode=0 Oct 30 19:41:22.843062 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:22.842999 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" event={"ID":"7456b22a-9c80-4fbb-9236-08eed4067902","Type":"ContainerDied","Data":"c8ad6e743a0419c65ea38f517284ca2c155a5070c5ab8d0c8824502908afc6de"} Oct 30 19:41:23.118771 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.118694 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:41:23.211906 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.211867 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-home\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-home\") pod \"7456b22a-9c80-4fbb-9236-08eed4067902\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " Oct 30 19:41:23.212130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.212019 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7456b22a-9c80-4fbb-9236-08eed4067902-tls-certs\" (UniqueName: \"kubernetes.io/secret/7456b22a-9c80-4fbb-9236-08eed4067902-tls-certs\") pod \"7456b22a-9c80-4fbb-9236-08eed4067902\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " Oct 30 19:41:23.212130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.212108 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7456b22a-9c80-4fbb-9236-08eed4067902-kube-api-access-d8bzt\" (UniqueName: \"kubernetes.io/projected/7456b22a-9c80-4fbb-9236-08eed4067902-kube-api-access-d8bzt\") pod \"7456b22a-9c80-4fbb-9236-08eed4067902\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " Oct 30 19:41:23.212263 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.212180 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-model-cache\") pod \"7456b22a-9c80-4fbb-9236-08eed4067902\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " Oct 30 19:41:23.212263 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.212243 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-home" pod "7456b22a-9c80-4fbb-9236-08eed4067902" (UID: "7456b22a-9c80-4fbb-9236-08eed4067902"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:23.212360 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.212317 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-dshm\") pod \"7456b22a-9c80-4fbb-9236-08eed4067902\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " Oct 30 19:41:23.212482 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.212464 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-tmp-dir\") pod \"7456b22a-9c80-4fbb-9236-08eed4067902\" (UID: \"7456b22a-9c80-4fbb-9236-08eed4067902\") " Oct 30 19:41:23.212540 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.212476 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-model-cache" pod "7456b22a-9c80-4fbb-9236-08eed4067902" (UID: "7456b22a-9c80-4fbb-9236-08eed4067902"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:23.212679 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.212658 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-tmp-dir" pod "7456b22a-9c80-4fbb-9236-08eed4067902" (UID: "7456b22a-9c80-4fbb-9236-08eed4067902"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:23.213100 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.213078 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:23.213244 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.213101 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:23.213244 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.213115 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:23.214810 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.214780 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7456b22a-9c80-4fbb-9236-08eed4067902-kube-api-access-d8bzt" pod "7456b22a-9c80-4fbb-9236-08eed4067902" (UID: "7456b22a-9c80-4fbb-9236-08eed4067902"). InnerVolumeSpecName "kube-api-access-d8bzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:23.214934 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.214909 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7456b22a-9c80-4fbb-9236-08eed4067902-tls-certs" pod "7456b22a-9c80-4fbb-9236-08eed4067902" (UID: "7456b22a-9c80-4fbb-9236-08eed4067902"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:41:23.215074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.215054 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-dshm" pod "7456b22a-9c80-4fbb-9236-08eed4067902" (UID: "7456b22a-9c80-4fbb-9236-08eed4067902"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:23.341148 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.339411 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7456b22a-9c80-4fbb-9236-08eed4067902-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:23.341148 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.339449 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8bzt\" (UniqueName: \"kubernetes.io/projected/7456b22a-9c80-4fbb-9236-08eed4067902-kube-api-access-d8bzt\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:23.341148 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.339466 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7456b22a-9c80-4fbb-9236-08eed4067902-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:23.500420 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.500321 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4"] Oct 30 19:41:23.852163 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.852134 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" Oct 30 19:41:23.852163 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.852151 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf" event={"ID":"7456b22a-9c80-4fbb-9236-08eed4067902","Type":"ContainerDied","Data":"de3fffb7fc96b60ebddafeb226ec00468a982281333f56ecc329afcbc150e216"} Oct 30 19:41:23.852401 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.852206 5047 scope.go:122] "RemoveContainer" containerID="c8ad6e743a0419c65ea38f517284ca2c155a5070c5ab8d0c8824502908afc6de" Oct 30 19:41:23.899940 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.899903 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf"] Oct 30 19:41:23.905255 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:23.905215 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9c8a2a/test-model-9c8a2a-kserve-5fcc7f6d87-9wgmf"] Oct 30 19:41:25.051781 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:25.051744 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7456b22a-9c80-4fbb-9236-08eed4067902" path="/var/lib/kubelet/pods/7456b22a-9c80-4fbb-9236-08eed4067902/volumes" Oct 30 19:41:32.846467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:32.846434 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" Oct 30 19:41:32.846932 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:32.846510 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" Oct 30 19:41:34.410367 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:34.410329 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk"] Oct 30 19:41:35.670895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.670856 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz"] Oct 30 19:41:35.671932 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.671907 5047 cpu_manager.go:414] "containerMap: removing container" podUID="7456b22a-9c80-4fbb-9236-08eed4067902" containerName="main" Oct 30 19:41:35.671932 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.671932 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7456b22a-9c80-4fbb-9236-08eed4067902" containerName="main" Oct 30 19:41:35.672183 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.672168 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="7456b22a-9c80-4fbb-9236-08eed4067902" containerName="main" Oct 30 19:41:35.675923 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.675903 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:41:35.690997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.690964 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e3504b-serving-cert\"" Oct 30 19:41:35.707554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.707517 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz"] Oct 30 19:41:35.714940 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.712415 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e689ed-675a-4ff5-a379-e2d7e751eb78-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz\" (UID: \"33e689ed-675a-4ff5-a379-e2d7e751eb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:41:35.715402 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.715303 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9qxk\" (UniqueName: \"kubernetes.io/projected/33e689ed-675a-4ff5-a379-e2d7e751eb78-kube-api-access-f9qxk\") pod \"maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz\" (UID: \"33e689ed-675a-4ff5-a379-e2d7e751eb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:41:35.715789 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.715765 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e689ed-675a-4ff5-a379-e2d7e751eb78-metrics-certs\") pod \"maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz\" (UID: \"33e689ed-675a-4ff5-a379-e2d7e751eb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:41:35.716118 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.716097 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e689ed-675a-4ff5-a379-e2d7e751eb78-maas-api-tls\") pod \"maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz\" (UID: \"33e689ed-675a-4ff5-a379-e2d7e751eb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:41:35.818201 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.818159 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e689ed-675a-4ff5-a379-e2d7e751eb78-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz\" (UID: \"33e689ed-675a-4ff5-a379-e2d7e751eb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:41:35.818435 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.818228 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9qxk\" (UniqueName: \"kubernetes.io/projected/33e689ed-675a-4ff5-a379-e2d7e751eb78-kube-api-access-f9qxk\") pod \"maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz\" (UID: \"33e689ed-675a-4ff5-a379-e2d7e751eb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:41:35.818435 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.818272 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e689ed-675a-4ff5-a379-e2d7e751eb78-metrics-certs\") pod \"maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz\" (UID: \"33e689ed-675a-4ff5-a379-e2d7e751eb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:41:35.818435 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.818363 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e689ed-675a-4ff5-a379-e2d7e751eb78-maas-api-tls\") pod \"maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz\" (UID: \"33e689ed-675a-4ff5-a379-e2d7e751eb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:41:35.818616 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:35.818520 5047 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-serving-cert: secret "maas-api-e2e-inf-b-e3504b-serving-cert" not found Oct 30 19:41:35.818616 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:35.818585 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33e689ed-675a-4ff5-a379-e2d7e751eb78-maas-api-tls podName:33e689ed-675a-4ff5-a379-e2d7e751eb78 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:36.318567502 +0000 UTC m=+2249.393022839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33e689ed-675a-4ff5-a379-e2d7e751eb78-maas-api-tls") pod "maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" (UID: "33e689ed-675a-4ff5-a379-e2d7e751eb78") : secret "maas-api-e2e-inf-b-e3504b-serving-cert" not found Oct 30 19:41:35.819042 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.818976 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e689ed-675a-4ff5-a379-e2d7e751eb78-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz\" (UID: \"33e689ed-675a-4ff5-a379-e2d7e751eb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:41:35.820969 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.820945 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e689ed-675a-4ff5-a379-e2d7e751eb78-metrics-certs\") pod \"maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz\" (UID: \"33e689ed-675a-4ff5-a379-e2d7e751eb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:41:35.838414 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:35.838385 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9qxk\" (UniqueName: \"kubernetes.io/projected/33e689ed-675a-4ff5-a379-e2d7e751eb78-kube-api-access-f9qxk\") pod \"maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz\" (UID: \"33e689ed-675a-4ff5-a379-e2d7e751eb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:41:36.326778 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:36.326742 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e689ed-675a-4ff5-a379-e2d7e751eb78-maas-api-tls\") pod \"maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz\" (UID: \"33e689ed-675a-4ff5-a379-e2d7e751eb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:41:36.329443 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:36.329417 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e689ed-675a-4ff5-a379-e2d7e751eb78-maas-api-tls\") pod \"maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz\" (UID: \"33e689ed-675a-4ff5-a379-e2d7e751eb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:41:36.601697 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:36.597350 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:41:36.933054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:36.933022 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz"] Oct 30 19:41:36.933753 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:41:36.933722 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e689ed_675a_4ff5_a379_e2d7e751eb78.slice/crio-8e1789ce5274f8f5f8ef2f7b1d2df204a286554675583a61749a6f1988a41031 WatchSource:0}: Error finding container 8e1789ce5274f8f5f8ef2f7b1d2df204a286554675583a61749a6f1988a41031: Status 404 returned error can't find the container with id 8e1789ce5274f8f5f8ef2f7b1d2df204a286554675583a61749a6f1988a41031 Oct 30 19:41:37.133919 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:37.133880 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" event={"ID":"33e689ed-675a-4ff5-a379-e2d7e751eb78","Type":"ContainerStarted","Data":"8e1789ce5274f8f5f8ef2f7b1d2df204a286554675583a61749a6f1988a41031"} Oct 30 19:41:38.107294 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.107252 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" event={"ID":"33e689ed-675a-4ff5-a379-e2d7e751eb78","Type":"ContainerStarted","Data":"a3873884bdfed552f9c326ea36015aa0511f27cc3bf1c0e7c8e11c72a1c9d2c3"} Oct 30 19:41:38.107742 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.107471 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:41:38.148787 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.148728 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" podStartSLOduration=2.6693010839999998 podStartE2EDuration="3.148713159s" totalImagesPullingTime="479.412075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:38.145198653 +0000 UTC m=+2251.219654016" watchObservedRunningTime="2026-10-30 19:41:38.148713159 +0000 UTC m=+2251.223168518" Oct 30 19:41:38.527746 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.527652 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" podUID="6dbadedf-fc49-43d8-82af-d7024864cc1f" containerName="main" containerID="cri-o://31e1534c761c4fb91e5d1412374d62bb8e3785131381e234439920cf5a6503ac" gracePeriod=45 Oct 30 19:41:38.826421 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.826395 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:41:38.901954 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.901916 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-dshm\") pod \"6dbadedf-fc49-43d8-82af-d7024864cc1f\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " Oct 30 19:41:38.902232 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.902211 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-home\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-home\") pod \"6dbadedf-fc49-43d8-82af-d7024864cc1f\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " Oct 30 19:41:38.902343 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.902284 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-tmp-dir\") pod \"6dbadedf-fc49-43d8-82af-d7024864cc1f\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " Oct 30 19:41:38.902407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.902368 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbadedf-fc49-43d8-82af-d7024864cc1f-kube-api-access-gl4qs\" (UniqueName: \"kubernetes.io/projected/6dbadedf-fc49-43d8-82af-d7024864cc1f-kube-api-access-gl4qs\") pod \"6dbadedf-fc49-43d8-82af-d7024864cc1f\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " Oct 30 19:41:38.902407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.902397 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dbadedf-fc49-43d8-82af-d7024864cc1f-tls-certs\" (UniqueName: \"kubernetes.io/secret/6dbadedf-fc49-43d8-82af-d7024864cc1f-tls-certs\") pod \"6dbadedf-fc49-43d8-82af-d7024864cc1f\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " Oct 30 19:41:38.902523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.902434 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-model-cache\") pod \"6dbadedf-fc49-43d8-82af-d7024864cc1f\" (UID: \"6dbadedf-fc49-43d8-82af-d7024864cc1f\") " Oct 30 19:41:38.902585 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.902562 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-home" pod "6dbadedf-fc49-43d8-82af-d7024864cc1f" (UID: "6dbadedf-fc49-43d8-82af-d7024864cc1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:38.902867 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.902840 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-model-cache" pod "6dbadedf-fc49-43d8-82af-d7024864cc1f" (UID: "6dbadedf-fc49-43d8-82af-d7024864cc1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:38.902916 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.902861 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-tmp-dir" pod "6dbadedf-fc49-43d8-82af-d7024864cc1f" (UID: "6dbadedf-fc49-43d8-82af-d7024864cc1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:38.903330 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.903299 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:38.903429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.903332 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:38.903429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.903349 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:38.905596 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.905563 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-dshm" pod "6dbadedf-fc49-43d8-82af-d7024864cc1f" (UID: "6dbadedf-fc49-43d8-82af-d7024864cc1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:38.905722 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.905694 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbadedf-fc49-43d8-82af-d7024864cc1f-kube-api-access-gl4qs" pod "6dbadedf-fc49-43d8-82af-d7024864cc1f" (UID: "6dbadedf-fc49-43d8-82af-d7024864cc1f"). InnerVolumeSpecName "kube-api-access-gl4qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:38.905859 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:38.905842 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dbadedf-fc49-43d8-82af-d7024864cc1f-tls-certs" pod "6dbadedf-fc49-43d8-82af-d7024864cc1f" (UID: "6dbadedf-fc49-43d8-82af-d7024864cc1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:41:39.005622 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:39.005579 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dbadedf-fc49-43d8-82af-d7024864cc1f-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:39.005622 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:39.005616 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl4qs\" (UniqueName: \"kubernetes.io/projected/6dbadedf-fc49-43d8-82af-d7024864cc1f-kube-api-access-gl4qs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:39.005914 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:39.005632 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dbadedf-fc49-43d8-82af-d7024864cc1f-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:39.116548 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:39.116515 5047 generic.go:356] "Generic (PLEG): container finished" podID="6dbadedf-fc49-43d8-82af-d7024864cc1f" containerID="31e1534c761c4fb91e5d1412374d62bb8e3785131381e234439920cf5a6503ac" exitCode=0 Oct 30 19:41:39.117025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:39.116565 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" Oct 30 19:41:39.117025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:39.116601 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" event={"ID":"6dbadedf-fc49-43d8-82af-d7024864cc1f","Type":"ContainerDied","Data":"31e1534c761c4fb91e5d1412374d62bb8e3785131381e234439920cf5a6503ac"} Oct 30 19:41:39.117025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:39.116642 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4" event={"ID":"6dbadedf-fc49-43d8-82af-d7024864cc1f","Type":"ContainerDied","Data":"e3f0077e8dfef05d8a2a21d74adb6f811edc929432a2bcc27847a305a959040c"} Oct 30 19:41:39.117025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:39.116660 5047 scope.go:122] "RemoveContainer" containerID="31e1534c761c4fb91e5d1412374d62bb8e3785131381e234439920cf5a6503ac" Oct 30 19:41:39.127416 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:39.127393 5047 scope.go:122] "RemoveContainer" containerID="31e1534c761c4fb91e5d1412374d62bb8e3785131381e234439920cf5a6503ac" Oct 30 19:41:39.127722 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:39.127702 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e1534c761c4fb91e5d1412374d62bb8e3785131381e234439920cf5a6503ac\": container with ID starting with 31e1534c761c4fb91e5d1412374d62bb8e3785131381e234439920cf5a6503ac not found: ID does not exist" containerID="31e1534c761c4fb91e5d1412374d62bb8e3785131381e234439920cf5a6503ac" Oct 30 19:41:39.127766 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:39.127730 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e1534c761c4fb91e5d1412374d62bb8e3785131381e234439920cf5a6503ac"} err="failed to get container status \"31e1534c761c4fb91e5d1412374d62bb8e3785131381e234439920cf5a6503ac\": rpc error: code = NotFound desc = could not find container \"31e1534c761c4fb91e5d1412374d62bb8e3785131381e234439920cf5a6503ac\": container with ID starting with 31e1534c761c4fb91e5d1412374d62bb8e3785131381e234439920cf5a6503ac not found: ID does not exist" Oct 30 19:41:39.162560 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:39.162528 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4"] Oct 30 19:41:39.166271 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:39.166242 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-19f646/test-model-19f646-kserve-6cdc876db7-b5rm4"] Oct 30 19:41:39.410939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:39.410899 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" podUID="dbd8e9c4-fdaa-45ff-a214-cac92b05c139" containerName="payload-pre-processing" containerID="cri-o://5f8c95daa048950bd30fd9344865277a92405977e82aec179b0ddea0259fa0ce" gracePeriod=25 Oct 30 19:41:39.602871 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:39.602821 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:41:41.052262 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:41.052222 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dbadedf-fc49-43d8-82af-d7024864cc1f" path="/var/lib/kubelet/pods/6dbadedf-fc49-43d8-82af-d7024864cc1f/volumes" Oct 30 19:41:44.122939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:44.122910 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:41:48.237131 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:48.237091 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" podUID="dbd8e9c4-fdaa-45ff-a214-cac92b05c139" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:41:50.513170 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.513127 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5"] Oct 30 19:41:50.514928 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.514894 5047 cpu_manager.go:414] "containerMap: removing container" podUID="6dbadedf-fc49-43d8-82af-d7024864cc1f" containerName="main" Oct 30 19:41:50.514928 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.514928 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbadedf-fc49-43d8-82af-d7024864cc1f" containerName="main" Oct 30 19:41:50.515219 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.515147 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbadedf-fc49-43d8-82af-d7024864cc1f" containerName="main" Oct 30 19:41:50.521445 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.521415 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.544829 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.544791 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5"] Oct 30 19:41:50.549057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.548968 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5d92fa\"/\"openshift-service-ca.crt\"" Oct 30 19:41:50.549216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.549073 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5d92fa\"/\"default-dockercfg-j824g\"" Oct 30 19:41:50.549216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.549098 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5d92fa\"/\"kube-root-ca.crt\"" Oct 30 19:41:50.549321 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.549233 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5d92fa\"/\"test-model-5d92fa-kserve-self-signed-certs\"" Oct 30 19:41:50.615179 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.615135 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-tmp-dir\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.615374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.615254 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d8d155-8327-4233-9fae-0d11e447bb0f-tls-certs\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.615374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.615303 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-dshm\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.615451 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.615382 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl5zj\" (UniqueName: \"kubernetes.io/projected/71d8d155-8327-4233-9fae-0d11e447bb0f-kube-api-access-xl5zj\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.615510 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.615498 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-home\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.615663 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.615650 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-model-cache\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.716980 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.716935 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-model-cache\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.717208 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.717073 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-tmp-dir\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.717208 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.717119 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d8d155-8327-4233-9fae-0d11e447bb0f-tls-certs\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.717208 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.717147 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-dshm\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.717208 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.717192 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl5zj\" (UniqueName: \"kubernetes.io/projected/71d8d155-8327-4233-9fae-0d11e447bb0f-kube-api-access-xl5zj\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.717493 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.717301 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-home\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.717546 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.717489 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-model-cache\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.717639 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.717603 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-tmp-dir\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.717728 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.717699 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-home\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.719570 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.719543 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-dshm\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.719855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.719837 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d8d155-8327-4233-9fae-0d11e447bb0f-tls-certs\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.736799 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.736757 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl5zj\" (UniqueName: \"kubernetes.io/projected/71d8d155-8327-4233-9fae-0d11e447bb0f-kube-api-access-xl5zj\") pod \"test-model-5d92fa-kserve-74579d55b9-tnzj5\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:50.847082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.837001 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm"] Oct 30 19:41:50.847082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.837289 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" podUID="a12e1654-8408-4422-8f28-37dbb4a77d49" containerName="maas-api" containerID="cri-o://41b53d90605bc7ce76ddf51737987692818b1733dca0fab8651fb86fc8c63161" gracePeriod=30 Oct 30 19:41:50.855586 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:50.855550 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:51.339578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.339544 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5"] Oct 30 19:41:51.436908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.436872 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:41:51.499551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.499514 5047 generic.go:356] "Generic (PLEG): container finished" podID="a12e1654-8408-4422-8f28-37dbb4a77d49" containerID="41b53d90605bc7ce76ddf51737987692818b1733dca0fab8651fb86fc8c63161" exitCode=0 Oct 30 19:41:51.499755 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.499737 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" Oct 30 19:41:51.499913 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.499890 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" event={"ID":"a12e1654-8408-4422-8f28-37dbb4a77d49","Type":"ContainerDied","Data":"41b53d90605bc7ce76ddf51737987692818b1733dca0fab8651fb86fc8c63161"} Oct 30 19:41:51.499966 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.499951 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm" event={"ID":"a12e1654-8408-4422-8f28-37dbb4a77d49","Type":"ContainerDied","Data":"088bea272ce13b5037ea9687769aef9432938fefac5536f4ce9fede1769ca7e7"} Oct 30 19:41:51.500066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.499985 5047 scope.go:122] "RemoveContainer" containerID="41b53d90605bc7ce76ddf51737987692818b1733dca0fab8651fb86fc8c63161" Oct 30 19:41:51.502569 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.502538 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" event={"ID":"71d8d155-8327-4233-9fae-0d11e447bb0f","Type":"ContainerStarted","Data":"74c2657dc3c8dbb7c25b5687a1e6c10a4b068907298d71d1ff6cad5ec9b200f6"} Oct 30 19:41:51.502705 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.502578 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" event={"ID":"71d8d155-8327-4233-9fae-0d11e447bb0f","Type":"ContainerStarted","Data":"028c5fe5647ef93802690ab53f73d7125f100d55951f5917a9506e7d8c4efdc6"} Oct 30 19:41:51.509275 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.509239 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a12e1654-8408-4422-8f28-37dbb4a77d49-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12e1654-8408-4422-8f28-37dbb4a77d49-metrics-certs\") pod \"a12e1654-8408-4422-8f28-37dbb4a77d49\" (UID: \"a12e1654-8408-4422-8f28-37dbb4a77d49\") " Oct 30 19:41:51.510030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.509418 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a12e1654-8408-4422-8f28-37dbb4a77d49-kube-api-access-86snq\" (UniqueName: \"kubernetes.io/projected/a12e1654-8408-4422-8f28-37dbb4a77d49-kube-api-access-86snq\") pod \"a12e1654-8408-4422-8f28-37dbb4a77d49\" (UID: \"a12e1654-8408-4422-8f28-37dbb4a77d49\") " Oct 30 19:41:51.510030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.509683 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a12e1654-8408-4422-8f28-37dbb4a77d49-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12e1654-8408-4422-8f28-37dbb4a77d49-maas-api-tls\") pod \"a12e1654-8408-4422-8f28-37dbb4a77d49\" (UID: \"a12e1654-8408-4422-8f28-37dbb4a77d49\") " Oct 30 19:41:51.510030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.509745 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a12e1654-8408-4422-8f28-37dbb4a77d49-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12e1654-8408-4422-8f28-37dbb4a77d49-openshift-service-ca\") pod \"a12e1654-8408-4422-8f28-37dbb4a77d49\" (UID: \"a12e1654-8408-4422-8f28-37dbb4a77d49\") " Oct 30 19:41:51.510893 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.510864 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12e1654-8408-4422-8f28-37dbb4a77d49-openshift-service-ca" pod "a12e1654-8408-4422-8f28-37dbb4a77d49" (UID: "a12e1654-8408-4422-8f28-37dbb4a77d49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:51.514319 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.514279 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a12e1654-8408-4422-8f28-37dbb4a77d49-metrics-certs" pod "a12e1654-8408-4422-8f28-37dbb4a77d49" (UID: "a12e1654-8408-4422-8f28-37dbb4a77d49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:41:51.517280 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.517254 5047 scope.go:122] "RemoveContainer" containerID="41b53d90605bc7ce76ddf51737987692818b1733dca0fab8651fb86fc8c63161" Oct 30 19:41:51.517805 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:51.517783 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b53d90605bc7ce76ddf51737987692818b1733dca0fab8651fb86fc8c63161\": container with ID starting with 41b53d90605bc7ce76ddf51737987692818b1733dca0fab8651fb86fc8c63161 not found: ID does not exist" containerID="41b53d90605bc7ce76ddf51737987692818b1733dca0fab8651fb86fc8c63161" Oct 30 19:41:51.517895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.517817 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b53d90605bc7ce76ddf51737987692818b1733dca0fab8651fb86fc8c63161"} err="failed to get container status \"41b53d90605bc7ce76ddf51737987692818b1733dca0fab8651fb86fc8c63161\": rpc error: code = NotFound desc = could not find container \"41b53d90605bc7ce76ddf51737987692818b1733dca0fab8651fb86fc8c63161\": container with ID starting with 41b53d90605bc7ce76ddf51737987692818b1733dca0fab8651fb86fc8c63161 not found: ID does not exist" Oct 30 19:41:51.524219 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.524184 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a12e1654-8408-4422-8f28-37dbb4a77d49-maas-api-tls" pod "a12e1654-8408-4422-8f28-37dbb4a77d49" (UID: "a12e1654-8408-4422-8f28-37dbb4a77d49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:41:51.526147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.526060 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12e1654-8408-4422-8f28-37dbb4a77d49-kube-api-access-86snq" pod "a12e1654-8408-4422-8f28-37dbb4a77d49" (UID: "a12e1654-8408-4422-8f28-37dbb4a77d49"). InnerVolumeSpecName "kube-api-access-86snq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:51.608043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.607963 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" podStartSLOduration=1.607946631 podStartE2EDuration="1.607946631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:51.602927462 +0000 UTC m=+2264.677382821" watchObservedRunningTime="2026-10-30 19:41:51.607946631 +0000 UTC m=+2264.682401990" Oct 30 19:41:51.612124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.612094 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86snq\" (UniqueName: \"kubernetes.io/projected/a12e1654-8408-4422-8f28-37dbb4a77d49-kube-api-access-86snq\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:51.612124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.612126 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12e1654-8408-4422-8f28-37dbb4a77d49-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:51.612331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.612138 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12e1654-8408-4422-8f28-37dbb4a77d49-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:51.612331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.612150 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12e1654-8408-4422-8f28-37dbb4a77d49-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:51.874126 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.869319 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:41:51.913484 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.913441 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm"] Oct 30 19:41:51.974051 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:51.973993 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8e6b67-75bb8494b6-thdcm"] Oct 30 19:41:53.051363 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:53.051327 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a12e1654-8408-4422-8f28-37dbb4a77d49" path="/var/lib/kubelet/pods/a12e1654-8408-4422-8f28-37dbb4a77d49/volumes" Oct 30 19:41:53.244293 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:53.244256 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw"] Oct 30 19:41:53.244534 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:53.244510 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" podUID="9859e88e-b269-42ac-8647-0bf70d7080cb" containerName="istio-proxy" containerID="cri-o://7b668b4cf4b747eb55dcf1e6bcf0e21419004f22cb478f94b7af161b10ee6eba" gracePeriod=30 Oct 30 19:41:54.426561 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.426529 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz"] Oct 30 19:41:54.427402 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.427383 5047 cpu_manager.go:414] "containerMap: removing container" podUID="a12e1654-8408-4422-8f28-37dbb4a77d49" containerName="maas-api" Oct 30 19:41:54.427478 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.427403 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a12e1654-8408-4422-8f28-37dbb4a77d49" containerName="maas-api" Oct 30 19:41:54.427561 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.427551 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="a12e1654-8408-4422-8f28-37dbb4a77d49" containerName="maas-api" Oct 30 19:41:54.433124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.433098 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.449142 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.446917 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f6cef-openshift-default-dockercfg-bflkp\"" Oct 30 19:41:54.469132 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.468961 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz"] Oct 30 19:41:54.598547 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.598508 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-workload-certs\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.598963 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.598920 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istiod-ca-cert\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.599277 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.599159 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-credential-socket\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.599434 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.599417 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-data\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.600058 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.600033 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-token\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.600509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.600400 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-envoy\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.605049 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.600677 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-workload-socket\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.605385 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.605362 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-podinfo\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.605868 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.605838 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh6td\" (UniqueName: \"kubernetes.io/projected/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-kube-api-access-dh6td\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.718101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.717286 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-credential-socket\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.718101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.717347 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-data\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.718101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.717449 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-token\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.718101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.717506 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-envoy\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.718101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.717551 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-workload-socket\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.718101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.717605 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-podinfo\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.718101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.717742 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6td\" (UniqueName: \"kubernetes.io/projected/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-kube-api-access-dh6td\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.718101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.717817 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-workload-certs\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.718101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.717916 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istiod-ca-cert\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.721449 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.718971 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istiod-ca-cert\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.721449 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.719299 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-credential-socket\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.721449 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.719535 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-data\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.730235 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.730200 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-workload-socket\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.733601 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.730993 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-workload-certs\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.784034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.772369 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh6td\" (UniqueName: \"kubernetes.io/projected/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-kube-api-access-dh6td\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.784034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.772544 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-podinfo\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.784034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.772839 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-envoy\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:54.799040 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:54.798575 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-token\") pod \"e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:55.068084 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:55.067990 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:55.401647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:55.401510 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz"] Oct 30 19:41:55.412727 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:55.412676 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:41:55.412912 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:55.412876 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:41:55.412989 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:55.412958 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:41:55.648865 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:55.648775 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" event={"ID":"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760","Type":"ContainerStarted","Data":"927e7a7ecdb4a82d5cb71d61feaf75a51eda445e695fd3b3d6e59fdd033f2782"} Oct 30 19:41:55.648865 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:55.648835 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" event={"ID":"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760","Type":"ContainerStarted","Data":"2eba252084590efbf9ec46495c0b371c8ecbdffa69b52b7a18fa8b79c2d89a7c"} Oct 30 19:41:55.698783 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:55.698715 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" podStartSLOduration=1.6986933610000001 podStartE2EDuration="1.698693361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:55.69732253 +0000 UTC m=+2268.771777890" watchObservedRunningTime="2026-10-30 19:41:55.698693361 +0000 UTC m=+2268.773148721" Oct 30 19:41:56.068958 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:56.068867 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:56.075256 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:56.075228 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:56.657108 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:56.657076 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:56.658249 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:56.658227 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:41:58.235145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.235101 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" podUID="dbd8e9c4-fdaa-45ff-a214-cac92b05c139" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:41:58.558109 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.558080 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:41:58.633055 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.633019 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-workload-socket\") pod \"9859e88e-b269-42ac-8647-0bf70d7080cb\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " Oct 30 19:41:58.633248 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.633083 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-data\") pod \"9859e88e-b269-42ac-8647-0bf70d7080cb\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " Oct 30 19:41:58.633248 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.633185 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-workload-certs\") pod \"9859e88e-b269-42ac-8647-0bf70d7080cb\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " Oct 30 19:41:58.633361 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.633329 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-podinfo\") pod \"9859e88e-b269-42ac-8647-0bf70d7080cb\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " Oct 30 19:41:58.633424 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.633387 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9859e88e-b269-42ac-8647-0bf70d7080cb-kube-api-access-rmcww\" (UniqueName: \"kubernetes.io/projected/9859e88e-b269-42ac-8647-0bf70d7080cb-kube-api-access-rmcww\") pod \"9859e88e-b269-42ac-8647-0bf70d7080cb\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " Oct 30 19:41:58.633474 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.633423 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-credential-socket\") pod \"9859e88e-b269-42ac-8647-0bf70d7080cb\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " Oct 30 19:41:58.633474 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.633381 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-workload-socket" pod "9859e88e-b269-42ac-8647-0bf70d7080cb" (UID: "9859e88e-b269-42ac-8647-0bf70d7080cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:58.633578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.633520 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9859e88e-b269-42ac-8647-0bf70d7080cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9859e88e-b269-42ac-8647-0bf70d7080cb-istiod-ca-cert\") pod \"9859e88e-b269-42ac-8647-0bf70d7080cb\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " Oct 30 19:41:58.633578 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.633547 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-token\" (UniqueName: \"kubernetes.io/projected/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-token\") pod \"9859e88e-b269-42ac-8647-0bf70d7080cb\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " Oct 30 19:41:58.633677 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.633657 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-envoy\") pod \"9859e88e-b269-42ac-8647-0bf70d7080cb\" (UID: \"9859e88e-b269-42ac-8647-0bf70d7080cb\") " Oct 30 19:41:58.634667 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.633821 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-data" pod "9859e88e-b269-42ac-8647-0bf70d7080cb" (UID: "9859e88e-b269-42ac-8647-0bf70d7080cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:58.634667 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.634198 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9859e88e-b269-42ac-8647-0bf70d7080cb-istiod-ca-cert" pod "9859e88e-b269-42ac-8647-0bf70d7080cb" (UID: "9859e88e-b269-42ac-8647-0bf70d7080cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:58.634667 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.634345 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9859e88e-b269-42ac-8647-0bf70d7080cb-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:58.634667 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.634365 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:58.634667 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.634366 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-credential-socket" pod "9859e88e-b269-42ac-8647-0bf70d7080cb" (UID: "9859e88e-b269-42ac-8647-0bf70d7080cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:58.634667 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.634378 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:58.634667 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.634517 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-workload-certs" pod "9859e88e-b269-42ac-8647-0bf70d7080cb" (UID: "9859e88e-b269-42ac-8647-0bf70d7080cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:58.636904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.636874 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9859e88e-b269-42ac-8647-0bf70d7080cb-kube-api-access-rmcww" pod "9859e88e-b269-42ac-8647-0bf70d7080cb" (UID: "9859e88e-b269-42ac-8647-0bf70d7080cb"). InnerVolumeSpecName "kube-api-access-rmcww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:58.637147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.637118 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-envoy" pod "9859e88e-b269-42ac-8647-0bf70d7080cb" (UID: "9859e88e-b269-42ac-8647-0bf70d7080cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:58.637653 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.637618 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-token" pod "9859e88e-b269-42ac-8647-0bf70d7080cb" (UID: "9859e88e-b269-42ac-8647-0bf70d7080cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:58.638732 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.638708 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-podinfo" pod "9859e88e-b269-42ac-8647-0bf70d7080cb" (UID: "9859e88e-b269-42ac-8647-0bf70d7080cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:41:58.675340 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.675300 5047 generic.go:356] "Generic (PLEG): container finished" podID="9859e88e-b269-42ac-8647-0bf70d7080cb" containerID="7b668b4cf4b747eb55dcf1e6bcf0e21419004f22cb478f94b7af161b10ee6eba" exitCode=0 Oct 30 19:41:58.675482 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.675347 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" Oct 30 19:41:58.675482 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.675372 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" event={"ID":"9859e88e-b269-42ac-8647-0bf70d7080cb","Type":"ContainerDied","Data":"7b668b4cf4b747eb55dcf1e6bcf0e21419004f22cb478f94b7af161b10ee6eba"} Oct 30 19:41:58.675482 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.675425 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw" event={"ID":"9859e88e-b269-42ac-8647-0bf70d7080cb","Type":"ContainerDied","Data":"73d308d4a422c18be57182661036607dc59e745779836adafd757a0851137573"} Oct 30 19:41:58.675482 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.675447 5047 scope.go:122] "RemoveContainer" containerID="7b668b4cf4b747eb55dcf1e6bcf0e21419004f22cb478f94b7af161b10ee6eba" Oct 30 19:41:58.688318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.688291 5047 scope.go:122] "RemoveContainer" containerID="f3126db0e1961ffb3fa4ed39c1beecef6c7018289741a6b340968eebcfb38bb2" Oct 30 19:41:58.699364 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.699336 5047 scope.go:122] "RemoveContainer" containerID="7b668b4cf4b747eb55dcf1e6bcf0e21419004f22cb478f94b7af161b10ee6eba" Oct 30 19:41:58.699720 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:58.699696 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b668b4cf4b747eb55dcf1e6bcf0e21419004f22cb478f94b7af161b10ee6eba\": container with ID starting with 7b668b4cf4b747eb55dcf1e6bcf0e21419004f22cb478f94b7af161b10ee6eba not found: ID does not exist" containerID="7b668b4cf4b747eb55dcf1e6bcf0e21419004f22cb478f94b7af161b10ee6eba" Oct 30 19:41:58.699814 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.699728 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b668b4cf4b747eb55dcf1e6bcf0e21419004f22cb478f94b7af161b10ee6eba"} err="failed to get container status \"7b668b4cf4b747eb55dcf1e6bcf0e21419004f22cb478f94b7af161b10ee6eba\": rpc error: code = NotFound desc = could not find container \"7b668b4cf4b747eb55dcf1e6bcf0e21419004f22cb478f94b7af161b10ee6eba\": container with ID starting with 7b668b4cf4b747eb55dcf1e6bcf0e21419004f22cb478f94b7af161b10ee6eba not found: ID does not exist" Oct 30 19:41:58.699814 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.699749 5047 scope.go:122] "RemoveContainer" containerID="f3126db0e1961ffb3fa4ed39c1beecef6c7018289741a6b340968eebcfb38bb2" Oct 30 19:41:58.700158 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:58.700129 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3126db0e1961ffb3fa4ed39c1beecef6c7018289741a6b340968eebcfb38bb2\": container with ID starting with f3126db0e1961ffb3fa4ed39c1beecef6c7018289741a6b340968eebcfb38bb2 not found: ID does not exist" containerID="f3126db0e1961ffb3fa4ed39c1beecef6c7018289741a6b340968eebcfb38bb2" Oct 30 19:41:58.700255 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.700170 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3126db0e1961ffb3fa4ed39c1beecef6c7018289741a6b340968eebcfb38bb2"} err="failed to get container status \"f3126db0e1961ffb3fa4ed39c1beecef6c7018289741a6b340968eebcfb38bb2\": rpc error: code = NotFound desc = could not find container \"f3126db0e1961ffb3fa4ed39c1beecef6c7018289741a6b340968eebcfb38bb2\": container with ID starting with f3126db0e1961ffb3fa4ed39c1beecef6c7018289741a6b340968eebcfb38bb2 not found: ID does not exist" Oct 30 19:41:58.722820 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.722789 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw"] Oct 30 19:41:58.730259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.730229 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a7c3b0-openshift-default-7d48cb547f-rnsrw"] Oct 30 19:41:58.736502 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.736472 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:58.736502 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.736499 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:58.736687 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.736508 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:58.736687 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.736523 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9859e88e-b269-42ac-8647-0bf70d7080cb-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:58.736687 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.736532 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmcww\" (UniqueName: \"kubernetes.io/projected/9859e88e-b269-42ac-8647-0bf70d7080cb-kube-api-access-rmcww\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:58.736687 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:58.736543 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9859e88e-b269-42ac-8647-0bf70d7080cb-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:59.168647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:59.168600 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9859e88e-b269-42ac-8647-0bf70d7080cb" path="/var/lib/kubelet/pods/9859e88e-b269-42ac-8647-0bf70d7080cb/volumes" Oct 30 19:41:59.636681 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:59.636655 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" Oct 30 19:41:59.686315 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:59.686233 5047 generic.go:356] "Generic (PLEG): container finished" podID="dbd8e9c4-fdaa-45ff-a214-cac92b05c139" containerID="5f8c95daa048950bd30fd9344865277a92405977e82aec179b0ddea0259fa0ce" exitCode=0 Oct 30 19:41:59.686315 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:59.686279 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" Oct 30 19:41:59.686553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:59.686312 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" event={"ID":"dbd8e9c4-fdaa-45ff-a214-cac92b05c139","Type":"ContainerDied","Data":"5f8c95daa048950bd30fd9344865277a92405977e82aec179b0ddea0259fa0ce"} Oct 30 19:41:59.686553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:59.686399 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk" event={"ID":"dbd8e9c4-fdaa-45ff-a214-cac92b05c139","Type":"ContainerDied","Data":"e2560a2920f2b79fb10c2efe35969a164be7aeacb3dbeccffda84ed3b96e3878"} Oct 30 19:41:59.686553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:59.686423 5047 scope.go:122] "RemoveContainer" containerID="5f8c95daa048950bd30fd9344865277a92405977e82aec179b0ddea0259fa0ce" Oct 30 19:41:59.700798 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:59.700777 5047 scope.go:122] "RemoveContainer" containerID="5f8c95daa048950bd30fd9344865277a92405977e82aec179b0ddea0259fa0ce" Oct 30 19:41:59.701154 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:41:59.701124 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f8c95daa048950bd30fd9344865277a92405977e82aec179b0ddea0259fa0ce\": container with ID starting with 5f8c95daa048950bd30fd9344865277a92405977e82aec179b0ddea0259fa0ce not found: ID does not exist" containerID="5f8c95daa048950bd30fd9344865277a92405977e82aec179b0ddea0259fa0ce" Oct 30 19:41:59.701239 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:59.701163 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8c95daa048950bd30fd9344865277a92405977e82aec179b0ddea0259fa0ce"} err="failed to get container status \"5f8c95daa048950bd30fd9344865277a92405977e82aec179b0ddea0259fa0ce\": rpc error: code = NotFound desc = could not find container \"5f8c95daa048950bd30fd9344865277a92405977e82aec179b0ddea0259fa0ce\": container with ID starting with 5f8c95daa048950bd30fd9344865277a92405977e82aec179b0ddea0259fa0ce not found: ID does not exist" Oct 30 19:41:59.759402 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:59.759350 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd8e9c4-fdaa-45ff-a214-cac92b05c139-kube-api-access-5x8q5\" (UniqueName: \"kubernetes.io/projected/dbd8e9c4-fdaa-45ff-a214-cac92b05c139-kube-api-access-5x8q5\") pod \"dbd8e9c4-fdaa-45ff-a214-cac92b05c139\" (UID: \"dbd8e9c4-fdaa-45ff-a214-cac92b05c139\") " Oct 30 19:41:59.759695 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:59.759659 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd8e9c4-fdaa-45ff-a214-cac92b05c139-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd8e9c4-fdaa-45ff-a214-cac92b05c139-plugins-config-volume\") pod \"dbd8e9c4-fdaa-45ff-a214-cac92b05c139\" (UID: \"dbd8e9c4-fdaa-45ff-a214-cac92b05c139\") " Oct 30 19:41:59.760053 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:59.759988 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd8e9c4-fdaa-45ff-a214-cac92b05c139-plugins-config-volume" pod "dbd8e9c4-fdaa-45ff-a214-cac92b05c139" (UID: "dbd8e9c4-fdaa-45ff-a214-cac92b05c139"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:59.760355 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:59.760338 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd8e9c4-fdaa-45ff-a214-cac92b05c139-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:41:59.761670 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:59.761647 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd8e9c4-fdaa-45ff-a214-cac92b05c139-kube-api-access-5x8q5" pod "dbd8e9c4-fdaa-45ff-a214-cac92b05c139" (UID: "dbd8e9c4-fdaa-45ff-a214-cac92b05c139"). InnerVolumeSpecName "kube-api-access-5x8q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:59.862138 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:41:59.862095 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x8q5\" (UniqueName: \"kubernetes.io/projected/dbd8e9c4-fdaa-45ff-a214-cac92b05c139-kube-api-access-5x8q5\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:42:00.032564 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.032529 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk"] Oct 30 19:42:00.038425 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.038394 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c8a2a-6d87cf6c-rwgrk"] Oct 30 19:42:00.067046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.066993 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr"] Oct 30 19:42:00.067901 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.067880 5047 cpu_manager.go:414] "containerMap: removing container" podUID="dbd8e9c4-fdaa-45ff-a214-cac92b05c139" containerName="payload-pre-processing" Oct 30 19:42:00.067901 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.067901 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd8e9c4-fdaa-45ff-a214-cac92b05c139" containerName="payload-pre-processing" Oct 30 19:42:00.068074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.067922 5047 cpu_manager.go:414] "containerMap: removing container" podUID="9859e88e-b269-42ac-8647-0bf70d7080cb" containerName="istio-proxy" Oct 30 19:42:00.068074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.067927 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9859e88e-b269-42ac-8647-0bf70d7080cb" containerName="istio-proxy" Oct 30 19:42:00.068074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.067936 5047 cpu_manager.go:414] "containerMap: removing container" podUID="9859e88e-b269-42ac-8647-0bf70d7080cb" containerName="istio-proxy" Oct 30 19:42:00.068074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.067940 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9859e88e-b269-42ac-8647-0bf70d7080cb" containerName="istio-proxy" Oct 30 19:42:00.068269 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.068085 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd8e9c4-fdaa-45ff-a214-cac92b05c139" containerName="payload-pre-processing" Oct 30 19:42:00.068269 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.068097 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="9859e88e-b269-42ac-8647-0bf70d7080cb" containerName="istio-proxy" Oct 30 19:42:00.068269 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.068104 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="9859e88e-b269-42ac-8647-0bf70d7080cb" containerName="istio-proxy" Oct 30 19:42:00.073435 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.073411 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" Oct 30 19:42:00.089371 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.089340 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1f6cef\"" Oct 30 19:42:00.089544 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.089464 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1f6cef-dockercfg-gl4pp\"" Oct 30 19:42:00.096558 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.096530 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr"] Oct 30 19:42:00.140971 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.140937 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz"] Oct 30 19:42:00.146037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.145997 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" Oct 30 19:42:00.161819 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.161784 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz"] Oct 30 19:42:00.167242 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.167204 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr\" (UID: \"bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" Oct 30 19:42:00.167819 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.167792 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc8ss\" (UniqueName: \"kubernetes.io/projected/bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5-kube-api-access-cc8ss\") pod \"payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr\" (UID: \"bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" Oct 30 19:42:00.269251 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.269157 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc8ss\" (UniqueName: \"kubernetes.io/projected/bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5-kube-api-access-cc8ss\") pod \"payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr\" (UID: \"bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" Oct 30 19:42:00.269444 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.269260 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr\" (UID: \"bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" Oct 30 19:42:00.269444 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.269284 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smv5f\" (UniqueName: \"kubernetes.io/projected/313dd358-14e7-4172-b9cb-3594dc0f4e7f-kube-api-access-smv5f\") pod \"payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz\" (UID: \"313dd358-14e7-4172-b9cb-3594dc0f4e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" Oct 30 19:42:00.269444 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.269427 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313dd358-14e7-4172-b9cb-3594dc0f4e7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz\" (UID: \"313dd358-14e7-4172-b9cb-3594dc0f4e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" Oct 30 19:42:00.270082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.270062 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr\" (UID: \"bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" Oct 30 19:42:00.286485 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.286456 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc8ss\" (UniqueName: \"kubernetes.io/projected/bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5-kube-api-access-cc8ss\") pod \"payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr\" (UID: \"bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" Oct 30 19:42:00.383040 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.380514 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smv5f\" (UniqueName: \"kubernetes.io/projected/313dd358-14e7-4172-b9cb-3594dc0f4e7f-kube-api-access-smv5f\") pod \"payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz\" (UID: \"313dd358-14e7-4172-b9cb-3594dc0f4e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" Oct 30 19:42:00.383040 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.380743 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313dd358-14e7-4172-b9cb-3594dc0f4e7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz\" (UID: \"313dd358-14e7-4172-b9cb-3594dc0f4e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" Oct 30 19:42:00.383040 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.382273 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313dd358-14e7-4172-b9cb-3594dc0f4e7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz\" (UID: \"313dd358-14e7-4172-b9cb-3594dc0f4e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" Oct 30 19:42:00.402773 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.399354 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" Oct 30 19:42:00.420280 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.416746 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smv5f\" (UniqueName: \"kubernetes.io/projected/313dd358-14e7-4172-b9cb-3594dc0f4e7f-kube-api-access-smv5f\") pod \"payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz\" (UID: \"313dd358-14e7-4172-b9cb-3594dc0f4e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" Oct 30 19:42:00.469039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.467319 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" Oct 30 19:42:00.856231 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.856191 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:42:00.869887 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.869855 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:42:00.974759 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.974724 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr"] Oct 30 19:42:00.974900 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:42:00.974756 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedb0dd5_4c08_4e9f_ae37_17ee9a39a2f5.slice/crio-b7112467f386e4eb6fe8d6b5ac921eaecace6b92416665cdec315cb88129cfef WatchSource:0}: Error finding container b7112467f386e4eb6fe8d6b5ac921eaecace6b92416665cdec315cb88129cfef: Status 404 returned error can't find the container with id b7112467f386e4eb6fe8d6b5ac921eaecace6b92416665cdec315cb88129cfef Oct 30 19:42:00.981917 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:00.981896 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz"] Oct 30 19:42:00.983254 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:42:00.983221 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313dd358_14e7_4172_b9cb_3594dc0f4e7f.slice/crio-ebbda6a40eb6ffd273bf6e59531385e52c60305b6ef35a509b4ea880a8e09df0 WatchSource:0}: Error finding container ebbda6a40eb6ffd273bf6e59531385e52c60305b6ef35a509b4ea880a8e09df0: Status 404 returned error can't find the container with id ebbda6a40eb6ffd273bf6e59531385e52c60305b6ef35a509b4ea880a8e09df0 Oct 30 19:42:01.044957 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.044918 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h"] Oct 30 19:42:01.045216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.045189 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" podUID="22ab4a50-e3a3-4312-8de3-4c9656b724de" containerName="istio-proxy" containerID="cri-o://4b1b09424cf99cc1f976febdd1ab41ef1b30b1b430b77e8af4b4c568aff4fb01" gracePeriod=30 Oct 30 19:42:01.074436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.074397 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd8e9c4-fdaa-45ff-a214-cac92b05c139" path="/var/lib/kubelet/pods/dbd8e9c4-fdaa-45ff-a214-cac92b05c139/volumes" Oct 30 19:42:01.095435 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.095405 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld"] Oct 30 19:42:01.101600 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.101579 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:42:01.144635 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.139289 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-f44f74-serving-cert\"" Oct 30 19:42:01.151833 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.151801 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld"] Oct 30 19:42:01.161520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.161488 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f350d91-ec30-4852-aa89-ad093cb6606a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld\" (UID: \"9f350d91-ec30-4852-aa89-ad093cb6606a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:42:01.161717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.161531 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f350d91-ec30-4852-aa89-ad093cb6606a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld\" (UID: \"9f350d91-ec30-4852-aa89-ad093cb6606a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:42:01.161800 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.161713 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f350d91-ec30-4852-aa89-ad093cb6606a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld\" (UID: \"9f350d91-ec30-4852-aa89-ad093cb6606a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:42:01.161862 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.161818 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc7kn\" (UniqueName: \"kubernetes.io/projected/9f350d91-ec30-4852-aa89-ad093cb6606a-kube-api-access-sc7kn\") pod \"maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld\" (UID: \"9f350d91-ec30-4852-aa89-ad093cb6606a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:42:01.263682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.263584 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f350d91-ec30-4852-aa89-ad093cb6606a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld\" (UID: \"9f350d91-ec30-4852-aa89-ad093cb6606a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:42:01.263682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.263680 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc7kn\" (UniqueName: \"kubernetes.io/projected/9f350d91-ec30-4852-aa89-ad093cb6606a-kube-api-access-sc7kn\") pod \"maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld\" (UID: \"9f350d91-ec30-4852-aa89-ad093cb6606a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:42:01.263891 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.263839 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f350d91-ec30-4852-aa89-ad093cb6606a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld\" (UID: \"9f350d91-ec30-4852-aa89-ad093cb6606a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:42:01.263891 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.263865 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f350d91-ec30-4852-aa89-ad093cb6606a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld\" (UID: \"9f350d91-ec30-4852-aa89-ad093cb6606a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:42:01.264397 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.264369 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f350d91-ec30-4852-aa89-ad093cb6606a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld\" (UID: \"9f350d91-ec30-4852-aa89-ad093cb6606a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:42:01.266513 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.266489 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f350d91-ec30-4852-aa89-ad093cb6606a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld\" (UID: \"9f350d91-ec30-4852-aa89-ad093cb6606a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:42:01.266588 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.266531 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f350d91-ec30-4852-aa89-ad093cb6606a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld\" (UID: \"9f350d91-ec30-4852-aa89-ad093cb6606a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:42:01.281423 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.281389 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc7kn\" (UniqueName: \"kubernetes.io/projected/9f350d91-ec30-4852-aa89-ad093cb6606a-kube-api-access-sc7kn\") pod \"maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld\" (UID: \"9f350d91-ec30-4852-aa89-ad093cb6606a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:42:01.498034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.487940 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:42:01.710264 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.710229 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" event={"ID":"bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5","Type":"ContainerStarted","Data":"5d804544b322c0ed609cd511ad0520c8dbbec82473205a6fa60b81dfcfe63384"} Oct 30 19:42:01.710471 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.710323 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" event={"ID":"bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5","Type":"ContainerStarted","Data":"b7112467f386e4eb6fe8d6b5ac921eaecace6b92416665cdec315cb88129cfef"} Oct 30 19:42:01.710471 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.710376 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" Oct 30 19:42:01.712177 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.712148 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" event={"ID":"313dd358-14e7-4172-b9cb-3594dc0f4e7f","Type":"ContainerStarted","Data":"66c087809cb66c6882225218b8844fbb926c373583222388af34933bb7b2c072"} Oct 30 19:42:01.712317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.712201 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" event={"ID":"313dd358-14e7-4172-b9cb-3594dc0f4e7f","Type":"ContainerStarted","Data":"ebbda6a40eb6ffd273bf6e59531385e52c60305b6ef35a509b4ea880a8e09df0"} Oct 30 19:42:01.712317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.712277 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" Oct 30 19:42:01.729317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.729245 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:42:01.756475 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.756410 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" podStartSLOduration=1.756394209 podStartE2EDuration="1.756394209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:01.745572087 +0000 UTC m=+2274.820027447" watchObservedRunningTime="2026-10-30 19:42:01.756394209 +0000 UTC m=+2274.830849562" Oct 30 19:42:01.784304 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:01.784244 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" podStartSLOduration=1.784225782 podStartE2EDuration="1.784225782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:01.781501964 +0000 UTC m=+2274.855957324" watchObservedRunningTime="2026-10-30 19:42:01.784225782 +0000 UTC m=+2274.858681144" Oct 30 19:42:02.013196 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:02.013153 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld"] Oct 30 19:42:02.015616 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:42:02.015454 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f350d91_ec30_4852_aa89_ad093cb6606a.slice/crio-97f7b9daeeb5ce1420a19040b1e54be3a8f1abf974412cd020fcab438452dfaf WatchSource:0}: Error finding container 97f7b9daeeb5ce1420a19040b1e54be3a8f1abf974412cd020fcab438452dfaf: Status 404 returned error can't find the container with id 97f7b9daeeb5ce1420a19040b1e54be3a8f1abf974412cd020fcab438452dfaf Oct 30 19:42:02.458297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:02.458253 5047 patch_prober.go:29] interesting pod/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:42:02.458442 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:02.458322 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" podUID="22ab4a50-e3a3-4312-8de3-4c9656b724de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:42:02.758311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:02.754526 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" event={"ID":"9f350d91-ec30-4852-aa89-ad093cb6606a","Type":"ContainerStarted","Data":"128323fa0fa0abe3e3d21feddc9675a7f0a3341b3963a682162c7c1d924fd81f"} Oct 30 19:42:02.758311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:02.754606 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" event={"ID":"9f350d91-ec30-4852-aa89-ad093cb6606a","Type":"ContainerStarted","Data":"97f7b9daeeb5ce1420a19040b1e54be3a8f1abf974412cd020fcab438452dfaf"} Oct 30 19:42:02.758311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:02.755165 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:42:02.795129 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:02.795071 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" podStartSLOduration=1.445171865 podStartE2EDuration="1.795053166s" totalImagesPullingTime="349.881301ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:02.790219091 +0000 UTC m=+2275.864674450" watchObservedRunningTime="2026-10-30 19:42:02.795053166 +0000 UTC m=+2275.869508566" Oct 30 19:42:06.387616 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.387582 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:42:06.477034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.474780 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-credential-socket\") pod \"22ab4a50-e3a3-4312-8de3-4c9656b724de\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " Oct 30 19:42:06.477034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.474958 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-podinfo\") pod \"22ab4a50-e3a3-4312-8de3-4c9656b724de\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " Oct 30 19:42:06.477034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.474997 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-envoy\") pod \"22ab4a50-e3a3-4312-8de3-4c9656b724de\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " Oct 30 19:42:06.477034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.475059 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ab4a50-e3a3-4312-8de3-4c9656b724de-kube-api-access-4l8kv\" (UniqueName: \"kubernetes.io/projected/22ab4a50-e3a3-4312-8de3-4c9656b724de-kube-api-access-4l8kv\") pod \"22ab4a50-e3a3-4312-8de3-4c9656b724de\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " Oct 30 19:42:06.477034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.475093 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-workload-certs\") pod \"22ab4a50-e3a3-4312-8de3-4c9656b724de\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " Oct 30 19:42:06.477034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.475153 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22ab4a50-e3a3-4312-8de3-4c9656b724de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ab4a50-e3a3-4312-8de3-4c9656b724de-istiod-ca-cert\") pod \"22ab4a50-e3a3-4312-8de3-4c9656b724de\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " Oct 30 19:42:06.477034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.475201 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-workload-socket\") pod \"22ab4a50-e3a3-4312-8de3-4c9656b724de\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " Oct 30 19:42:06.477034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.475268 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-token\" (UniqueName: \"kubernetes.io/projected/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-token\") pod \"22ab4a50-e3a3-4312-8de3-4c9656b724de\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " Oct 30 19:42:06.477034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.475399 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-data\") pod \"22ab4a50-e3a3-4312-8de3-4c9656b724de\" (UID: \"22ab4a50-e3a3-4312-8de3-4c9656b724de\") " Oct 30 19:42:06.478350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.477410 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-workload-socket" pod "22ab4a50-e3a3-4312-8de3-4c9656b724de" (UID: "22ab4a50-e3a3-4312-8de3-4c9656b724de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:06.478350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.477493 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-credential-socket" pod "22ab4a50-e3a3-4312-8de3-4c9656b724de" (UID: "22ab4a50-e3a3-4312-8de3-4c9656b724de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:06.478350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.477917 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-podinfo" pod "22ab4a50-e3a3-4312-8de3-4c9656b724de" (UID: "22ab4a50-e3a3-4312-8de3-4c9656b724de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:42:06.478522 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.478450 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ab4a50-e3a3-4312-8de3-4c9656b724de-istiod-ca-cert" pod "22ab4a50-e3a3-4312-8de3-4c9656b724de" (UID: "22ab4a50-e3a3-4312-8de3-4c9656b724de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:06.478522 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.478515 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-workload-certs" pod "22ab4a50-e3a3-4312-8de3-4c9656b724de" (UID: "22ab4a50-e3a3-4312-8de3-4c9656b724de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:06.478633 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.478576 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-data" pod "22ab4a50-e3a3-4312-8de3-4c9656b724de" (UID: "22ab4a50-e3a3-4312-8de3-4c9656b724de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:06.480099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.480076 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ab4a50-e3a3-4312-8de3-4c9656b724de-kube-api-access-4l8kv" pod "22ab4a50-e3a3-4312-8de3-4c9656b724de" (UID: "22ab4a50-e3a3-4312-8de3-4c9656b724de"). InnerVolumeSpecName "kube-api-access-4l8kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:06.480997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.480972 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-envoy" pod "22ab4a50-e3a3-4312-8de3-4c9656b724de" (UID: "22ab4a50-e3a3-4312-8de3-4c9656b724de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:06.481186 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.481168 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-token" pod "22ab4a50-e3a3-4312-8de3-4c9656b724de" (UID: "22ab4a50-e3a3-4312-8de3-4c9656b724de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:06.578197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.578161 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.578197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.578191 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.578197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.578201 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l8kv\" (UniqueName: \"kubernetes.io/projected/22ab4a50-e3a3-4312-8de3-4c9656b724de-kube-api-access-4l8kv\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.578412 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.578210 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.578412 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.578221 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ab4a50-e3a3-4312-8de3-4c9656b724de-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.578412 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.578229 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.578412 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.578237 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.578412 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.578244 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.578412 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.578252 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ab4a50-e3a3-4312-8de3-4c9656b724de-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:42:06.797954 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.797855 5047 generic.go:356] "Generic (PLEG): container finished" podID="22ab4a50-e3a3-4312-8de3-4c9656b724de" containerID="4b1b09424cf99cc1f976febdd1ab41ef1b30b1b430b77e8af4b4c568aff4fb01" exitCode=0 Oct 30 19:42:06.797954 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.797921 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" Oct 30 19:42:06.798232 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.797943 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" event={"ID":"22ab4a50-e3a3-4312-8de3-4c9656b724de","Type":"ContainerDied","Data":"4b1b09424cf99cc1f976febdd1ab41ef1b30b1b430b77e8af4b4c568aff4fb01"} Oct 30 19:42:06.798232 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.798035 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h" event={"ID":"22ab4a50-e3a3-4312-8de3-4c9656b724de","Type":"ContainerDied","Data":"1e225a12c9ecde6d81de9aff57019c518c43d49d7def7715beb78a2cb443a337"} Oct 30 19:42:06.798232 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.798070 5047 scope.go:122] "RemoveContainer" containerID="4b1b09424cf99cc1f976febdd1ab41ef1b30b1b430b77e8af4b4c568aff4fb01" Oct 30 19:42:06.820968 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.820939 5047 scope.go:122] "RemoveContainer" containerID="d3a04500d82cb2b769d0cfe5acc230928b4320dca934df79bf853ee2b17aaaae" Oct 30 19:42:06.832151 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.832126 5047 scope.go:122] "RemoveContainer" containerID="4b1b09424cf99cc1f976febdd1ab41ef1b30b1b430b77e8af4b4c568aff4fb01" Oct 30 19:42:06.832513 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:42:06.832483 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b1b09424cf99cc1f976febdd1ab41ef1b30b1b430b77e8af4b4c568aff4fb01\": container with ID starting with 4b1b09424cf99cc1f976febdd1ab41ef1b30b1b430b77e8af4b4c568aff4fb01 not found: ID does not exist" containerID="4b1b09424cf99cc1f976febdd1ab41ef1b30b1b430b77e8af4b4c568aff4fb01" Oct 30 19:42:06.832649 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.832518 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b1b09424cf99cc1f976febdd1ab41ef1b30b1b430b77e8af4b4c568aff4fb01"} err="failed to get container status \"4b1b09424cf99cc1f976febdd1ab41ef1b30b1b430b77e8af4b4c568aff4fb01\": rpc error: code = NotFound desc = could not find container \"4b1b09424cf99cc1f976febdd1ab41ef1b30b1b430b77e8af4b4c568aff4fb01\": container with ID starting with 4b1b09424cf99cc1f976febdd1ab41ef1b30b1b430b77e8af4b4c568aff4fb01 not found: ID does not exist" Oct 30 19:42:06.832649 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.832536 5047 scope.go:122] "RemoveContainer" containerID="d3a04500d82cb2b769d0cfe5acc230928b4320dca934df79bf853ee2b17aaaae" Oct 30 19:42:06.832827 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:42:06.832799 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3a04500d82cb2b769d0cfe5acc230928b4320dca934df79bf853ee2b17aaaae\": container with ID starting with d3a04500d82cb2b769d0cfe5acc230928b4320dca934df79bf853ee2b17aaaae not found: ID does not exist" containerID="d3a04500d82cb2b769d0cfe5acc230928b4320dca934df79bf853ee2b17aaaae" Oct 30 19:42:06.832872 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.832836 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3a04500d82cb2b769d0cfe5acc230928b4320dca934df79bf853ee2b17aaaae"} err="failed to get container status \"d3a04500d82cb2b769d0cfe5acc230928b4320dca934df79bf853ee2b17aaaae\": rpc error: code = NotFound desc = could not find container \"d3a04500d82cb2b769d0cfe5acc230928b4320dca934df79bf853ee2b17aaaae\": container with ID starting with d3a04500d82cb2b769d0cfe5acc230928b4320dca934df79bf853ee2b17aaaae not found: ID does not exist" Oct 30 19:42:06.866501 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.866459 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h"] Oct 30 19:42:06.870359 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:06.870325 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9c8a2a-openshift-default-6ccfcf5fd7-gd46h"] Oct 30 19:42:07.081345 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:07.081315 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ab4a50-e3a3-4312-8de3-4c9656b724de" path="/var/lib/kubelet/pods/22ab4a50-e3a3-4312-8de3-4c9656b724de/volumes" Oct 30 19:42:08.779490 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:08.778414 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:42:09.604071 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:42:09.604033 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:42:10.263571 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.263524 5047 scope.go:122] "RemoveContainer" containerID="3826a3cbb86d2ce0a39335345f6607e2237ce6469e594dcdd1cf3621f55a1cc0" Oct 30 19:42:10.674361 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.674328 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22"] Oct 30 19:42:10.675350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.675325 5047 cpu_manager.go:414] "containerMap: removing container" podUID="22ab4a50-e3a3-4312-8de3-4c9656b724de" containerName="istio-proxy" Oct 30 19:42:10.675350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.675349 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ab4a50-e3a3-4312-8de3-4c9656b724de" containerName="istio-proxy" Oct 30 19:42:10.675615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.675370 5047 cpu_manager.go:414] "containerMap: removing container" podUID="22ab4a50-e3a3-4312-8de3-4c9656b724de" containerName="istio-proxy" Oct 30 19:42:10.675615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.675378 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ab4a50-e3a3-4312-8de3-4c9656b724de" containerName="istio-proxy" Oct 30 19:42:10.675742 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.675648 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ab4a50-e3a3-4312-8de3-4c9656b724de" containerName="istio-proxy" Oct 30 19:42:10.675742 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.675669 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ab4a50-e3a3-4312-8de3-4c9656b724de" containerName="istio-proxy" Oct 30 19:42:10.688392 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.688085 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:10.702362 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.701601 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22"] Oct 30 19:42:10.702722 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.702645 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-469724\"/\"openshift-service-ca.crt\"" Oct 30 19:42:10.703095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.703053 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-469724\"/\"default-dockercfg-vj87j\"" Oct 30 19:42:10.712603 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.712564 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-469724\"/\"kube-root-ca.crt\"" Oct 30 19:42:10.712744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.712608 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-469724\"/\"test-model-469724-kserve-self-signed-certs\"" Oct 30 19:42:10.872420 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.872383 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a306b8ac-1021-4e38-be1c-aea68485bea3-tls-certs\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:10.872626 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.872606 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-dshm\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:10.872734 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.872717 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-model-cache\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:10.872768 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.872744 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-home\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:10.872827 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.872816 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9zzg\" (UniqueName: \"kubernetes.io/projected/a306b8ac-1021-4e38-be1c-aea68485bea3-kube-api-access-d9zzg\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:10.872872 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.872859 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-tmp-dir\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:10.981615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.978036 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-dshm\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:10.981615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.978179 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-model-cache\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:10.981615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.978208 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-home\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:10.981615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.978288 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9zzg\" (UniqueName: \"kubernetes.io/projected/a306b8ac-1021-4e38-be1c-aea68485bea3-kube-api-access-d9zzg\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:10.981615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.978319 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-tmp-dir\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:10.981615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.978386 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a306b8ac-1021-4e38-be1c-aea68485bea3-tls-certs\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:10.981615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.979488 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-home\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:10.981615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.980469 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-model-cache\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:10.981615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:10.980734 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-tmp-dir\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:11.079098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:11.078718 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9zzg\" (UniqueName: \"kubernetes.io/projected/a306b8ac-1021-4e38-be1c-aea68485bea3-kube-api-access-d9zzg\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:11.084552 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:11.082339 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-dshm\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:11.084552 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:11.082968 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a306b8ac-1021-4e38-be1c-aea68485bea3-tls-certs\") pod \"test-model-469724-kserve-d45bd94f9-2sc22\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:11.311017 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:11.310920 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:11.667593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:11.667552 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22"] Oct 30 19:42:11.669622 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:42:11.669587 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda306b8ac_1021_4e38_be1c_aea68485bea3.slice/crio-904d71f22c0d965922070ae9e9a839ec7973de6d997aa6a5c2e678993df13dc5 WatchSource:0}: Error finding container 904d71f22c0d965922070ae9e9a839ec7973de6d997aa6a5c2e678993df13dc5: Status 404 returned error can't find the container with id 904d71f22c0d965922070ae9e9a839ec7973de6d997aa6a5c2e678993df13dc5 Oct 30 19:42:11.881112 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:11.881076 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" event={"ID":"a306b8ac-1021-4e38-be1c-aea68485bea3","Type":"ContainerStarted","Data":"2af3c1e898d95b3fccd2c376f6e036d5b5eac281e721bf49d21a16e4829a359a"} Oct 30 19:42:11.881112 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:11.881127 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" event={"ID":"a306b8ac-1021-4e38-be1c-aea68485bea3","Type":"ContainerStarted","Data":"904d71f22c0d965922070ae9e9a839ec7973de6d997aa6a5c2e678993df13dc5"} Oct 30 19:42:11.917309 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:11.917162 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" podStartSLOduration=1.9171470149999998 podStartE2EDuration="1.917147015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:11.915052125 +0000 UTC m=+2284.989507478" watchObservedRunningTime="2026-10-30 19:42:11.917147015 +0000 UTC m=+2284.991602373" Oct 30 19:42:12.311884 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:12.311777 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:12.759356 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:12.759326 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" Oct 30 19:42:12.759553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:12.759404 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" Oct 30 19:42:13.692215 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.692179 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54"] Oct 30 19:42:13.698057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.698026 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.718349 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.718301 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54"] Oct 30 19:42:13.720739 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.720710 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-08bcfe\"/\"default-dockercfg-4s6qd\"" Oct 30 19:42:13.721604 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.721559 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-08bcfe\"/\"kube-root-ca.crt\"" Oct 30 19:42:13.721604 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.721588 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-08bcfe\"/\"test-model-08bcfe-kserve-self-signed-certs\"" Oct 30 19:42:13.721777 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.721739 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-08bcfe\"/\"openshift-service-ca.crt\"" Oct 30 19:42:13.730373 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.730337 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-model-cache\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.730550 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.730400 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkfht\" (UniqueName: \"kubernetes.io/projected/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-kube-api-access-xkfht\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.730615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.730556 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-tmp-dir\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.730615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.730592 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-dshm\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.730776 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.730652 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-tls-certs\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.730776 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.730681 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-home\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.832318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.832275 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-model-cache\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.832540 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.832380 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkfht\" (UniqueName: \"kubernetes.io/projected/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-kube-api-access-xkfht\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.833593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.832936 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-tmp-dir\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.833593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.833067 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-model-cache\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.833593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.833288 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-tmp-dir\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.833593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.833411 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-dshm\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.833593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.833471 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-tls-certs\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.833593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.833496 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-home\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.834196 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.834177 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-home\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.839146 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.836272 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-dshm\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.839146 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.836564 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-tls-certs\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:13.852759 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:13.852721 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkfht\" (UniqueName: \"kubernetes.io/projected/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-kube-api-access-xkfht\") pod \"test-model-08bcfe-kserve-74456787ff-25p54\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:14.024081 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:14.023284 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:14.439943 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:42:14.439875 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79c14c0f_27b4_4124_8d40_2e4a0ea85a6e.slice/crio-5db7a73025e5d20541f23e0bb1cab16ba4280dd545ea835fcd65412535bb1363 WatchSource:0}: Error finding container 5db7a73025e5d20541f23e0bb1cab16ba4280dd545ea835fcd65412535bb1363: Status 404 returned error can't find the container with id 5db7a73025e5d20541f23e0bb1cab16ba4280dd545ea835fcd65412535bb1363 Oct 30 19:42:14.507451 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:14.507393 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54"] Oct 30 19:42:14.909945 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:14.909912 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" event={"ID":"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e","Type":"ContainerStarted","Data":"af2323d6a7516e3afd54a21062af79043c10207f850030182b791a528f181ea7"} Oct 30 19:42:14.910448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:14.909958 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" event={"ID":"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e","Type":"ContainerStarted","Data":"5db7a73025e5d20541f23e0bb1cab16ba4280dd545ea835fcd65412535bb1363"} Oct 30 19:42:14.947729 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:14.947669 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" podStartSLOduration=1.947649773 podStartE2EDuration="1.947649773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:14.945055869 +0000 UTC m=+2288.019511226" watchObservedRunningTime="2026-10-30 19:42:14.947649773 +0000 UTC m=+2288.022105134" Oct 30 19:42:15.024715 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:15.024678 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:16.926827 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:16.926797 5047 generic.go:356] "Generic (PLEG): container finished" podID="299d2abb-3201-42ac-8f33-edc45040171a" containerID="000c8cc05ac04716cd74114a6bc3f4d2330f997caed5e8a45280bfb3001cbdb4" exitCode=0 Oct 30 19:42:16.927295 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:16.926872 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" event={"ID":"299d2abb-3201-42ac-8f33-edc45040171a","Type":"ContainerDied","Data":"000c8cc05ac04716cd74114a6bc3f4d2330f997caed5e8a45280bfb3001cbdb4"} Oct 30 19:42:16.927448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:16.927431 5047 scope.go:122] "RemoveContainer" containerID="000c8cc05ac04716cd74114a6bc3f4d2330f997caed5e8a45280bfb3001cbdb4" Oct 30 19:42:16.929098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:16.929071 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:42:16.929179 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:16.929142 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:42:16.929220 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:16.929177 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:42:17.947455 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:17.942068 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" event={"ID":"299d2abb-3201-42ac-8f33-edc45040171a","Type":"ContainerStarted","Data":"dcf227a913476a447a2d180c56fb0a4783471f0579bb2fab430cbadf9ca39fe2"} Oct 30 19:42:18.489839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:18.489803 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:42:18.491019 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:18.490974 5047 patch_prober.go:29] interesting pod/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 19:42:18.491159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:18.491035 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" podUID="299d2abb-3201-42ac-8f33-edc45040171a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 19:42:19.490362 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:19.490324 5047 patch_prober.go:29] interesting pod/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 19:42:19.490780 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:19.490400 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" podUID="299d2abb-3201-42ac-8f33-edc45040171a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 19:42:19.997031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:19.996683 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:42:20.200643 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.200599 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv"] Oct 30 19:42:20.213312 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.211686 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.230228 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.230194 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-09feeb\"/\"test-model-09feeb-kserve-self-signed-certs\"" Oct 30 19:42:20.230602 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.230572 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-09feeb\"/\"kube-root-ca.crt\"" Oct 30 19:42:20.257086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.244313 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv"] Oct 30 19:42:20.261410 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.261379 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-09feeb\"/\"default-dockercfg-cgdw8\"" Oct 30 19:42:20.261722 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.261706 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-09feeb\"/\"openshift-service-ca.crt\"" Oct 30 19:42:20.315828 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.315768 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-home\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.316033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.315846 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-tmp-dir\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.316033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.315972 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-model-cache\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.316175 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.316155 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj25q\" (UniqueName: \"kubernetes.io/projected/c183d261-57b7-4c90-817e-b4cedd105113-kube-api-access-qj25q\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.316234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.316218 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c183d261-57b7-4c90-817e-b4cedd105113-tls-certs\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.316412 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.316385 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-dshm\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.418718 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.418676 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-home\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.418718 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.418735 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-tmp-dir\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.418980 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.418806 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-model-cache\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.418980 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.418901 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj25q\" (UniqueName: \"kubernetes.io/projected/c183d261-57b7-4c90-817e-b4cedd105113-kube-api-access-qj25q\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.418980 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.418954 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c183d261-57b7-4c90-817e-b4cedd105113-tls-certs\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.419101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.419031 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-dshm\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.419224 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.419200 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-tmp-dir\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.441042 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.432810 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-model-cache\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.441747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.441579 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-home\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.476030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.475122 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj25q\" (UniqueName: \"kubernetes.io/projected/c183d261-57b7-4c90-817e-b4cedd105113-kube-api-access-qj25q\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.476030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.475223 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-dshm\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.476030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.475934 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c183d261-57b7-4c90-817e-b4cedd105113-tls-certs\") pod \"test-model-09feeb-kserve-595dcf7d45-lt9lv\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.525078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.524700 5047 patch_prober.go:29] interesting pod/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 19:42:20.525078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.524763 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" podUID="299d2abb-3201-42ac-8f33-edc45040171a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 19:42:20.573581 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.573546 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:20.970590 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:42:20.970490 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc183d261_57b7_4c90_817e_b4cedd105113.slice/crio-7dc57f846419c88eda7860a95ffd323408109f5d8a71e9284a4510274015be9f WatchSource:0}: Error finding container 7dc57f846419c88eda7860a95ffd323408109f5d8a71e9284a4510274015be9f: Status 404 returned error can't find the container with id 7dc57f846419c88eda7860a95ffd323408109f5d8a71e9284a4510274015be9f Oct 30 19:42:20.970815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:20.970785 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv"] Oct 30 19:42:21.013217 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:21.013147 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" event={"ID":"c183d261-57b7-4c90-817e-b4cedd105113","Type":"ContainerStarted","Data":"7dc57f846419c88eda7860a95ffd323408109f5d8a71e9284a4510274015be9f"} Oct 30 19:42:21.311376 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:21.311276 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:21.324811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:21.324773 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:21.490723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:21.490683 5047 patch_prober.go:29] interesting pod/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 19:42:21.490914 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:21.490759 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" podUID="299d2abb-3201-42ac-8f33-edc45040171a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 19:42:22.021935 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:22.021901 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" event={"ID":"c183d261-57b7-4c90-817e-b4cedd105113","Type":"ContainerStarted","Data":"74c4cbb5697a3d81058c7809061bc93d69ee51e645bded81a215ee2482de0746"} Oct 30 19:42:22.027132 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:22.027098 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv"] Oct 30 19:42:22.040804 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:22.037585 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:42:22.077478 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:22.077415 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" podStartSLOduration=2.077398419 podStartE2EDuration="2.077398419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:22.07258099 +0000 UTC m=+2295.147036394" watchObservedRunningTime="2026-10-30 19:42:22.077398419 +0000 UTC m=+2295.151853777" Oct 30 19:42:22.490418 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:22.490384 5047 patch_prober.go:29] interesting pod/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 19:42:22.490588 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:22.490440 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" podUID="299d2abb-3201-42ac-8f33-edc45040171a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 19:42:22.574581 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:22.574544 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:23.493456 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:23.493416 5047 patch_prober.go:29] interesting pod/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 19:42:23.494126 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:23.494089 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" podUID="299d2abb-3201-42ac-8f33-edc45040171a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 19:42:24.023761 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:24.023727 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:24.043445 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:24.043389 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:24.058400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:24.058367 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:42:24.495477 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:24.495442 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:42:24.496712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:24.496687 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:42:27.027663 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:27.027617 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" podUID="0dbb629f-7a48-4553-abc8-46ec8c370a35" containerName="payload-processing" containerID="cri-o://e23538c76db4ba34576b3ba5d19cf894c0ad11abce0464af482c6aaedf40414f" gracePeriod=25 Oct 30 19:42:29.220511 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:29.220463 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" podUID="0dbb629f-7a48-4553-abc8-46ec8c370a35" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:42:30.574473 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:30.574439 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:30.588164 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:30.588137 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:31.264453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:31.264420 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:42:39.220868 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:39.220824 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" podUID="0dbb629f-7a48-4553-abc8-46ec8c370a35" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:42:39.605567 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:42:39.605531 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:42:47.555363 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:47.551676 5047 generic.go:356] "Generic (PLEG): container finished" podID="0dbb629f-7a48-4553-abc8-46ec8c370a35" containerID="e23538c76db4ba34576b3ba5d19cf894c0ad11abce0464af482c6aaedf40414f" exitCode=0 Oct 30 19:42:47.555363 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:47.551761 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" event={"ID":"0dbb629f-7a48-4553-abc8-46ec8c370a35","Type":"ContainerDied","Data":"e23538c76db4ba34576b3ba5d19cf894c0ad11abce0464af482c6aaedf40414f"} Oct 30 19:42:47.581908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:47.581876 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" Oct 30 19:42:47.730326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:47.730287 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dbb629f-7a48-4553-abc8-46ec8c370a35-kube-api-access-5fq9d\" (UniqueName: \"kubernetes.io/projected/0dbb629f-7a48-4553-abc8-46ec8c370a35-kube-api-access-5fq9d\") pod \"0dbb629f-7a48-4553-abc8-46ec8c370a35\" (UID: \"0dbb629f-7a48-4553-abc8-46ec8c370a35\") " Oct 30 19:42:47.730536 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:47.730370 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dbb629f-7a48-4553-abc8-46ec8c370a35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbb629f-7a48-4553-abc8-46ec8c370a35-plugins-config-volume\") pod \"0dbb629f-7a48-4553-abc8-46ec8c370a35\" (UID: \"0dbb629f-7a48-4553-abc8-46ec8c370a35\") " Oct 30 19:42:47.730801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:47.730768 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dbb629f-7a48-4553-abc8-46ec8c370a35-plugins-config-volume" pod "0dbb629f-7a48-4553-abc8-46ec8c370a35" (UID: "0dbb629f-7a48-4553-abc8-46ec8c370a35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:47.731166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:47.731147 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbb629f-7a48-4553-abc8-46ec8c370a35-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:42:47.732758 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:47.732733 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbb629f-7a48-4553-abc8-46ec8c370a35-kube-api-access-5fq9d" pod "0dbb629f-7a48-4553-abc8-46ec8c370a35" (UID: "0dbb629f-7a48-4553-abc8-46ec8c370a35"). InnerVolumeSpecName "kube-api-access-5fq9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:47.833565 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:47.833524 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fq9d\" (UniqueName: \"kubernetes.io/projected/0dbb629f-7a48-4553-abc8-46ec8c370a35-kube-api-access-5fq9d\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:42:48.580970 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:48.580929 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" event={"ID":"0dbb629f-7a48-4553-abc8-46ec8c370a35","Type":"ContainerDied","Data":"7ca702c8ab4734fdd4d16543ad51954bce7d7271c1c5d178fb447ae5f8e6f607"} Oct 30 19:42:48.581583 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:48.581562 5047 scope.go:122] "RemoveContainer" containerID="e23538c76db4ba34576b3ba5d19cf894c0ad11abce0464af482c6aaedf40414f" Oct 30 19:42:48.581799 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:48.581445 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv" Oct 30 19:42:48.664095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:48.634781 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv"] Oct 30 19:42:48.669522 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:48.669492 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19f646-6f8dc65d78-7x6fv"] Oct 30 19:42:49.077356 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:49.077281 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dbb629f-7a48-4553-abc8-46ec8c370a35" path="/var/lib/kubelet/pods/0dbb629f-7a48-4553-abc8-46ec8c370a35/volumes" Oct 30 19:42:49.977759 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:49.977717 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp"] Oct 30 19:42:49.978286 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:49.977922 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" podUID="0b0c1a23-136f-4fc5-9c09-ab162557fb8f" containerName="maas-api" containerID="cri-o://a2157133950e09abbaa84af249277d5bb9bfd074e0573252369bf0c7f933bf5e" gracePeriod=30 Oct 30 19:42:50.714839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:50.714801 5047 generic.go:356] "Generic (PLEG): container finished" podID="0b0c1a23-136f-4fc5-9c09-ab162557fb8f" containerID="a2157133950e09abbaa84af249277d5bb9bfd074e0573252369bf0c7f933bf5e" exitCode=0 Oct 30 19:42:50.740726 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:50.715069 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" event={"ID":"0b0c1a23-136f-4fc5-9c09-ab162557fb8f","Type":"ContainerDied","Data":"a2157133950e09abbaa84af249277d5bb9bfd074e0573252369bf0c7f933bf5e"} Oct 30 19:42:50.802857 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:50.802819 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:42:50.872856 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:50.872823 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-openshift-service-ca\") pod \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\" (UID: \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\") " Oct 30 19:42:50.873171 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:50.873150 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-maas-api-tls\") pod \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\" (UID: \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\") " Oct 30 19:42:50.873288 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:50.873253 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-openshift-service-ca" pod "0b0c1a23-136f-4fc5-9c09-ab162557fb8f" (UID: "0b0c1a23-136f-4fc5-9c09-ab162557fb8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:50.873368 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:50.873352 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-metrics-certs\") pod \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\" (UID: \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\") " Oct 30 19:42:50.873474 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:50.873458 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-kube-api-access-h98jc\" (UniqueName: \"kubernetes.io/projected/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-kube-api-access-h98jc\") pod \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\" (UID: \"0b0c1a23-136f-4fc5-9c09-ab162557fb8f\") " Oct 30 19:42:50.874222 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:50.874200 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:42:50.876025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:50.875766 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-maas-api-tls" pod "0b0c1a23-136f-4fc5-9c09-ab162557fb8f" (UID: "0b0c1a23-136f-4fc5-9c09-ab162557fb8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:50.876025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:50.875849 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-metrics-certs" pod "0b0c1a23-136f-4fc5-9c09-ab162557fb8f" (UID: "0b0c1a23-136f-4fc5-9c09-ab162557fb8f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:50.876165 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:50.876113 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-kube-api-access-h98jc" pod "0b0c1a23-136f-4fc5-9c09-ab162557fb8f" (UID: "0b0c1a23-136f-4fc5-9c09-ab162557fb8f"). InnerVolumeSpecName "kube-api-access-h98jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:50.976343 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:50.976255 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:42:50.976571 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:50.976549 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:42:50.976778 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:50.976759 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h98jc\" (UniqueName: \"kubernetes.io/projected/0b0c1a23-136f-4fc5-9c09-ab162557fb8f-kube-api-access-h98jc\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:42:51.766551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:51.766514 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" event={"ID":"0b0c1a23-136f-4fc5-9c09-ab162557fb8f","Type":"ContainerDied","Data":"72b2f1681f374b7e5c88b3dd8476f85e2e1173c80d8bca460fc3ff22f12f8bc9"} Oct 30 19:42:51.766551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:51.766540 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp" Oct 30 19:42:51.767183 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:51.766582 5047 scope.go:122] "RemoveContainer" containerID="a2157133950e09abbaa84af249277d5bb9bfd074e0573252369bf0c7f933bf5e" Oct 30 19:42:51.861779 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:51.861743 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp"] Oct 30 19:42:51.866650 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:51.866618 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19f646-5f48b67666-zq2hp"] Oct 30 19:42:53.053752 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:53.053717 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b0c1a23-136f-4fc5-9c09-ab162557fb8f" path="/var/lib/kubelet/pods/0b0c1a23-136f-4fc5-9c09-ab162557fb8f/volumes" Oct 30 19:42:56.815955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:56.815918 5047 generic.go:356] "Generic (PLEG): container finished" podID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" containerID="fd3e459e68032a1e32b941284916d7d31a89b7d1df2d912944e62cf9201d5d6e" exitCode=0 Oct 30 19:42:56.816465 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:56.815992 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" event={"ID":"dec5c9c7-1994-4064-b2fe-ef9f288c53ba","Type":"ContainerDied","Data":"fd3e459e68032a1e32b941284916d7d31a89b7d1df2d912944e62cf9201d5d6e"} Oct 30 19:42:56.816552 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:56.816538 5047 scope.go:122] "RemoveContainer" containerID="fd3e459e68032a1e32b941284916d7d31a89b7d1df2d912944e62cf9201d5d6e" Oct 30 19:42:56.818444 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:56.818417 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:42:56.818522 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:56.818488 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:42:56.818560 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:56.818521 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:42:57.904737 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:57.904692 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" event={"ID":"dec5c9c7-1994-4064-b2fe-ef9f288c53ba","Type":"ContainerStarted","Data":"d312f4fab864e06fa964e693e03de323d06038b4fb3befd618b72b37625fdd08"} Oct 30 19:42:58.077778 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:58.076914 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw"] Oct 30 19:42:58.077778 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:58.077165 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" podUID="78be6f9f-292a-4259-bb67-0d1dd1365325" containerName="istio-proxy" containerID="cri-o://aa06c200c6b227b8e85d661ede2e97b3c2d54d48f59afac57d01b5d82bcdf62f" gracePeriod=30 Oct 30 19:42:58.231061 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:58.230923 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:42:58.231257 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:58.231133 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 19:42:58.231257 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:58.231187 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" podUID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 19:42:59.230831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.230793 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 19:42:59.231273 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.230871 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" podUID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 19:42:59.491089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.482477 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp"] Oct 30 19:42:59.491089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.483659 5047 cpu_manager.go:414] "containerMap: removing container" podUID="0dbb629f-7a48-4553-abc8-46ec8c370a35" containerName="payload-processing" Oct 30 19:42:59.491089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.483683 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dbb629f-7a48-4553-abc8-46ec8c370a35" containerName="payload-processing" Oct 30 19:42:59.491089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.483763 5047 cpu_manager.go:414] "containerMap: removing container" podUID="0b0c1a23-136f-4fc5-9c09-ab162557fb8f" containerName="maas-api" Oct 30 19:42:59.491089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.483771 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0c1a23-136f-4fc5-9c09-ab162557fb8f" containerName="maas-api" Oct 30 19:42:59.491089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.483932 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0c1a23-136f-4fc5-9c09-ab162557fb8f" containerName="maas-api" Oct 30 19:42:59.491089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.483952 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dbb629f-7a48-4553-abc8-46ec8c370a35" containerName="payload-processing" Oct 30 19:42:59.637031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.630193 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp"] Oct 30 19:42:59.637031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.630344 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.681038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.678171 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-095f55-openshift-default-dockercfg-ljw6s\"" Oct 30 19:42:59.695047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.689853 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-credential-socket\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.695047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.689907 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-envoy\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.695047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.689940 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-podinfo\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.695047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.690029 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-data\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.695047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.690211 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-workload-socket\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.695047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.690242 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea906da9-9a7c-4755-b5f1-2f6388875be2-istiod-ca-cert\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.695047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.690287 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-token\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.695047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.690371 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-workload-certs\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.695047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.690439 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d65tb\" (UniqueName: \"kubernetes.io/projected/ea906da9-9a7c-4755-b5f1-2f6388875be2-kube-api-access-d65tb\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.823735 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.823637 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-workload-socket\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.823993 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.823968 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea906da9-9a7c-4755-b5f1-2f6388875be2-istiod-ca-cert\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.824178 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.824162 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-token\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.824373 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.824357 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-workload-certs\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.824618 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.824587 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d65tb\" (UniqueName: \"kubernetes.io/projected/ea906da9-9a7c-4755-b5f1-2f6388875be2-kube-api-access-d65tb\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.824918 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.824890 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-credential-socket\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.825074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.825059 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-envoy\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.825249 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.825229 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-podinfo\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.825486 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.825460 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-workload-socket\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.826855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.826183 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea906da9-9a7c-4755-b5f1-2f6388875be2-istiod-ca-cert\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.826855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.826283 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-data\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.827473 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.827211 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-data\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.827473 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.827424 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-credential-socket\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.827923 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.827890 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-workload-certs\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.829071 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.828979 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-envoy\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.831221 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.830545 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-podinfo\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.877037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.874603 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d65tb\" (UniqueName: \"kubernetes.io/projected/ea906da9-9a7c-4755-b5f1-2f6388875be2-kube-api-access-d65tb\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:42:59.877663 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:42:59.877630 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-token\") pod \"e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:43:00.118154 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:00.117803 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:43:00.274296 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:00.274245 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 19:43:00.274778 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:00.274320 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" podUID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 19:43:00.476640 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:00.476594 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp"] Oct 30 19:43:00.477396 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:43:00.477363 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea906da9_9a7c_4755_b5f1_2f6388875be2.slice/crio-ef935aadaeedd8307417d686138367607c6c5d013bf4e6e87d2660b020c8f052 WatchSource:0}: Error finding container ef935aadaeedd8307417d686138367607c6c5d013bf4e6e87d2660b020c8f052: Status 404 returned error can't find the container with id ef935aadaeedd8307417d686138367607c6c5d013bf4e6e87d2660b020c8f052 Oct 30 19:43:00.481096 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:00.481035 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:43:00.481261 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:00.481237 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:43:00.481308 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:00.481296 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:43:00.933227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:00.933181 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" event={"ID":"ea906da9-9a7c-4755-b5f1-2f6388875be2","Type":"ContainerStarted","Data":"004460907acd6a0cfa6d797c7623ed7e37abdd83f0532b4b4862c59d34dbd984"} Oct 30 19:43:00.933420 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:00.933259 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" event={"ID":"ea906da9-9a7c-4755-b5f1-2f6388875be2","Type":"ContainerStarted","Data":"ef935aadaeedd8307417d686138367607c6c5d013bf4e6e87d2660b020c8f052"} Oct 30 19:43:00.988624 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:00.988561 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" podStartSLOduration=1.9885466040000002 podStartE2EDuration="1.988546604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:00.987236389 +0000 UTC m=+2334.061691749" watchObservedRunningTime="2026-10-30 19:43:00.988546604 +0000 UTC m=+2334.063001959" Oct 30 19:43:01.137031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:01.124975 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:43:01.137031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:01.125191 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 19:43:01.137031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:01.125243 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" podUID="ea906da9-9a7c-4755-b5f1-2f6388875be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 19:43:01.232715 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:01.232181 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 19:43:01.232715 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:01.232245 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" podUID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 19:43:01.730584 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:01.730524 5047 patch_prober.go:29] interesting pod/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:43:01.731113 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:01.730588 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" podUID="78be6f9f-292a-4259-bb67-0d1dd1365325" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:43:02.124405 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:02.124371 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:43:02.231392 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:02.231351 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 19:43:02.231563 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:02.231424 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" podUID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 19:43:02.974390 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:02.974354 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:43:02.975390 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:02.975365 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:43:03.232061 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.231949 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 19:43:03.232061 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.232030 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" podUID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 19:43:03.434729 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.434690 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:43:03.578886 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.578846 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-workload-socket\") pod \"78be6f9f-292a-4259-bb67-0d1dd1365325\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " Oct 30 19:43:03.579117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.578998 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-podinfo\") pod \"78be6f9f-292a-4259-bb67-0d1dd1365325\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " Oct 30 19:43:03.579117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.579034 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78be6f9f-292a-4259-bb67-0d1dd1365325-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78be6f9f-292a-4259-bb67-0d1dd1365325-istiod-ca-cert\") pod \"78be6f9f-292a-4259-bb67-0d1dd1365325\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " Oct 30 19:43:03.579117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.579086 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-credential-socket\") pod \"78be6f9f-292a-4259-bb67-0d1dd1365325\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " Oct 30 19:43:03.579323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.579133 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-envoy\") pod \"78be6f9f-292a-4259-bb67-0d1dd1365325\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " Oct 30 19:43:03.579323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.579163 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-workload-socket" pod "78be6f9f-292a-4259-bb67-0d1dd1365325" (UID: "78be6f9f-292a-4259-bb67-0d1dd1365325"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:03.579323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.579170 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-data\") pod \"78be6f9f-292a-4259-bb67-0d1dd1365325\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " Oct 30 19:43:03.579323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.579301 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-workload-certs\") pod \"78be6f9f-292a-4259-bb67-0d1dd1365325\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " Oct 30 19:43:03.579551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.579515 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-credential-socket" pod "78be6f9f-292a-4259-bb67-0d1dd1365325" (UID: "78be6f9f-292a-4259-bb67-0d1dd1365325"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:03.579650 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.579601 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-workload-certs" pod "78be6f9f-292a-4259-bb67-0d1dd1365325" (UID: "78be6f9f-292a-4259-bb67-0d1dd1365325"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:03.579723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.579619 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-data" pod "78be6f9f-292a-4259-bb67-0d1dd1365325" (UID: "78be6f9f-292a-4259-bb67-0d1dd1365325"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:03.579723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.579669 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-token\" (UniqueName: \"kubernetes.io/projected/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-token\") pod \"78be6f9f-292a-4259-bb67-0d1dd1365325\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " Oct 30 19:43:03.579723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.579709 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78be6f9f-292a-4259-bb67-0d1dd1365325-kube-api-access-gcp6g\" (UniqueName: \"kubernetes.io/projected/78be6f9f-292a-4259-bb67-0d1dd1365325-kube-api-access-gcp6g\") pod \"78be6f9f-292a-4259-bb67-0d1dd1365325\" (UID: \"78be6f9f-292a-4259-bb67-0d1dd1365325\") " Oct 30 19:43:03.579895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.579777 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78be6f9f-292a-4259-bb67-0d1dd1365325-istiod-ca-cert" pod "78be6f9f-292a-4259-bb67-0d1dd1365325" (UID: "78be6f9f-292a-4259-bb67-0d1dd1365325"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:03.580591 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.580562 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:43:03.580717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.580596 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78be6f9f-292a-4259-bb67-0d1dd1365325-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:43:03.580717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.580620 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:43:03.580717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.580636 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:43:03.580717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.580651 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:43:03.581692 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.581662 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-envoy" pod "78be6f9f-292a-4259-bb67-0d1dd1365325" (UID: "78be6f9f-292a-4259-bb67-0d1dd1365325"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:03.581990 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.581968 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-token" pod "78be6f9f-292a-4259-bb67-0d1dd1365325" (UID: "78be6f9f-292a-4259-bb67-0d1dd1365325"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:03.582220 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.582194 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-podinfo" pod "78be6f9f-292a-4259-bb67-0d1dd1365325" (UID: "78be6f9f-292a-4259-bb67-0d1dd1365325"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:43:03.582695 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.582670 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78be6f9f-292a-4259-bb67-0d1dd1365325-kube-api-access-gcp6g" pod "78be6f9f-292a-4259-bb67-0d1dd1365325" (UID: "78be6f9f-292a-4259-bb67-0d1dd1365325"). InnerVolumeSpecName "kube-api-access-gcp6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:03.682646 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.682599 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:43:03.682646 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.682641 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:43:03.682646 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.682656 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcp6g\" (UniqueName: \"kubernetes.io/projected/78be6f9f-292a-4259-bb67-0d1dd1365325-kube-api-access-gcp6g\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:43:03.682927 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.682670 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78be6f9f-292a-4259-bb67-0d1dd1365325-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:43:03.984296 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.984213 5047 generic.go:356] "Generic (PLEG): container finished" podID="78be6f9f-292a-4259-bb67-0d1dd1365325" containerID="aa06c200c6b227b8e85d661ede2e97b3c2d54d48f59afac57d01b5d82bcdf62f" exitCode=0 Oct 30 19:43:03.984296 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.984270 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" Oct 30 19:43:03.984787 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.984293 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" event={"ID":"78be6f9f-292a-4259-bb67-0d1dd1365325","Type":"ContainerDied","Data":"aa06c200c6b227b8e85d661ede2e97b3c2d54d48f59afac57d01b5d82bcdf62f"} Oct 30 19:43:03.984787 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.984357 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw" event={"ID":"78be6f9f-292a-4259-bb67-0d1dd1365325","Type":"ContainerDied","Data":"a0b86914ecc4be376f8a0da1d1f8d0fcbb9457c68e37b2c9655c7cd0ae3425ee"} Oct 30 19:43:03.984787 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.984390 5047 scope.go:122] "RemoveContainer" containerID="aa06c200c6b227b8e85d661ede2e97b3c2d54d48f59afac57d01b5d82bcdf62f" Oct 30 19:43:03.996839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:03.996809 5047 scope.go:122] "RemoveContainer" containerID="430f8a4b7e0569c133f6151c6449a8334ed5743667888b300e06850b869c9471" Oct 30 19:43:04.013561 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:04.013539 5047 scope.go:122] "RemoveContainer" containerID="aa06c200c6b227b8e85d661ede2e97b3c2d54d48f59afac57d01b5d82bcdf62f" Oct 30 19:43:04.013929 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:43:04.013903 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa06c200c6b227b8e85d661ede2e97b3c2d54d48f59afac57d01b5d82bcdf62f\": container with ID starting with aa06c200c6b227b8e85d661ede2e97b3c2d54d48f59afac57d01b5d82bcdf62f not found: ID does not exist" containerID="aa06c200c6b227b8e85d661ede2e97b3c2d54d48f59afac57d01b5d82bcdf62f" Oct 30 19:43:04.013987 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:04.013937 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa06c200c6b227b8e85d661ede2e97b3c2d54d48f59afac57d01b5d82bcdf62f"} err="failed to get container status \"aa06c200c6b227b8e85d661ede2e97b3c2d54d48f59afac57d01b5d82bcdf62f\": rpc error: code = NotFound desc = could not find container \"aa06c200c6b227b8e85d661ede2e97b3c2d54d48f59afac57d01b5d82bcdf62f\": container with ID starting with aa06c200c6b227b8e85d661ede2e97b3c2d54d48f59afac57d01b5d82bcdf62f not found: ID does not exist" Oct 30 19:43:04.013987 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:04.013958 5047 scope.go:122] "RemoveContainer" containerID="430f8a4b7e0569c133f6151c6449a8334ed5743667888b300e06850b869c9471" Oct 30 19:43:04.014344 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:43:04.014324 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"430f8a4b7e0569c133f6151c6449a8334ed5743667888b300e06850b869c9471\": container with ID starting with 430f8a4b7e0569c133f6151c6449a8334ed5743667888b300e06850b869c9471 not found: ID does not exist" containerID="430f8a4b7e0569c133f6151c6449a8334ed5743667888b300e06850b869c9471" Oct 30 19:43:04.014385 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:04.014354 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430f8a4b7e0569c133f6151c6449a8334ed5743667888b300e06850b869c9471"} err="failed to get container status \"430f8a4b7e0569c133f6151c6449a8334ed5743667888b300e06850b869c9471\": rpc error: code = NotFound desc = could not find container \"430f8a4b7e0569c133f6151c6449a8334ed5743667888b300e06850b869c9471\": container with ID starting with 430f8a4b7e0569c133f6151c6449a8334ed5743667888b300e06850b869c9471 not found: ID does not exist" Oct 30 19:43:04.039187 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:04.039150 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw"] Oct 30 19:43:04.057480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:04.057445 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-19f646-openshift-default-5bdfd85d65-qxmcw"] Oct 30 19:43:04.237223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:04.237144 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:43:04.237542 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:04.237524 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:43:04.238333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:04.238308 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:43:05.052702 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:05.052665 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78be6f9f-292a-4259-bb67-0d1dd1365325" path="/var/lib/kubelet/pods/78be6f9f-292a-4259-bb67-0d1dd1365325/volumes" Oct 30 19:43:06.648733 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.648690 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd"] Oct 30 19:43:06.649956 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.649930 5047 cpu_manager.go:414] "containerMap: removing container" podUID="78be6f9f-292a-4259-bb67-0d1dd1365325" containerName="istio-proxy" Oct 30 19:43:06.650032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.649959 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78be6f9f-292a-4259-bb67-0d1dd1365325" containerName="istio-proxy" Oct 30 19:43:06.650267 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.650250 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="78be6f9f-292a-4259-bb67-0d1dd1365325" containerName="istio-proxy" Oct 30 19:43:06.650305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.650283 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="78be6f9f-292a-4259-bb67-0d1dd1365325" containerName="istio-proxy" Oct 30 19:43:06.656284 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.656258 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.673414 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.673382 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd"] Oct 30 19:43:06.674917 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.674885 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e3504b\"/\"openshift-service-ca.crt\"" Oct 30 19:43:06.679790 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.679760 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e3504b\"/\"kube-root-ca.crt\"" Oct 30 19:43:06.679986 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.679826 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e3504b\"/\"test-model-e3504b-kserve-self-signed-certs\"" Oct 30 19:43:06.679986 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.679840 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e3504b\"/\"default-dockercfg-xqd67\"" Oct 30 19:43:06.737584 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.737542 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-dshm\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.737784 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.737720 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-home\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.737927 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.737907 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-tmp-dir\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.737992 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.737976 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4k9q\" (UniqueName: \"kubernetes.io/projected/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-kube-api-access-v4k9q\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.738170 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.738156 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-tls-certs\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.738204 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.738189 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-model-cache\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.839682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.839636 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-home\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.839879 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.839805 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-tmp-dir\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.839879 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.839862 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4k9q\" (UniqueName: \"kubernetes.io/projected/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-kube-api-access-v4k9q\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.839955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.839940 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-tls-certs\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.839994 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.839966 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-model-cache\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.840080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.839993 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-dshm\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.840116 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.840098 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-home\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.840228 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.840204 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-tmp-dir\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.840345 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.840326 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-model-cache\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.842362 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.842336 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-dshm\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.842745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.842722 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-tls-certs\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.858083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.857992 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4k9q\" (UniqueName: \"kubernetes.io/projected/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-kube-api-access-v4k9q\") pod \"test-model-e3504b-kserve-56656d9545-58dxd\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:06.975027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:06.974657 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:07.439918 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:07.439881 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd"] Oct 30 19:43:07.440114 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:43:07.439940 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c9e6db1_6c49_4d93_bdfb_85d984c8e9bc.slice/crio-8144a61699adc41be7e4a2482d8d1361fd204365c10e873366c752aa7764ebb8 WatchSource:0}: Error finding container 8144a61699adc41be7e4a2482d8d1361fd204365c10e873366c752aa7764ebb8: Status 404 returned error can't find the container with id 8144a61699adc41be7e4a2482d8d1361fd204365c10e873366c752aa7764ebb8 Oct 30 19:43:08.135378 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:08.135339 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" event={"ID":"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc","Type":"ContainerStarted","Data":"590a01d503bcc07f98ac6f81f3e0224c4be8b9d9676dc625155d7465221d9642"} Oct 30 19:43:08.135378 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:08.135385 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" event={"ID":"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc","Type":"ContainerStarted","Data":"8144a61699adc41be7e4a2482d8d1361fd204365c10e873366c752aa7764ebb8"} Oct 30 19:43:08.174285 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:08.174233 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" podStartSLOduration=2.17421668 podStartE2EDuration="2.17421668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:08.169438699 +0000 UTC m=+2341.243894059" watchObservedRunningTime="2026-10-30 19:43:08.17421668 +0000 UTC m=+2341.248672039" Oct 30 19:43:08.975645 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:08.975606 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:09.603889 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:43:09.603843 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:43:16.976191 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:16.976146 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:16.990278 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:16.990247 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:17.231532 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:17.231448 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:43:23.376102 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:23.376063 5047 generic.go:356] "Generic (PLEG): container finished" podID="2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" containerID="927e7a7ecdb4a82d5cb71d61feaf75a51eda445e695fd3b3d6e59fdd033f2782" exitCode=0 Oct 30 19:43:23.376589 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:23.376139 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" event={"ID":"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760","Type":"ContainerDied","Data":"927e7a7ecdb4a82d5cb71d61feaf75a51eda445e695fd3b3d6e59fdd033f2782"} Oct 30 19:43:23.376741 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:23.376722 5047 scope.go:122] "RemoveContainer" containerID="927e7a7ecdb4a82d5cb71d61feaf75a51eda445e695fd3b3d6e59fdd033f2782" Oct 30 19:43:23.378810 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:23.378777 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:43:23.378902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:23.378855 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:43:23.378902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:23.378893 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:43:24.398988 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:24.398941 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" event={"ID":"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760","Type":"ContainerStarted","Data":"80da37ab3e7a5b1fadd5702fc43322124898f3a08b717daba180eacc67b58932"} Oct 30 19:43:25.069447 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.069410 5047 patch_prober.go:29] interesting pod/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 19:43:25.069672 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.069512 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" podUID="2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 19:43:25.080225 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.080193 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:43:25.175333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.175300 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s"] Oct 30 19:43:25.176135 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.176119 5047 cpu_manager.go:414] "containerMap: removing container" podUID="78be6f9f-292a-4259-bb67-0d1dd1365325" containerName="istio-proxy" Oct 30 19:43:25.176189 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.176137 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78be6f9f-292a-4259-bb67-0d1dd1365325" containerName="istio-proxy" Oct 30 19:43:25.180378 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.180353 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.199366 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.199332 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a7b7bc\"/\"kube-root-ca.crt\"" Oct 30 19:43:25.199525 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.199390 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a7b7bc\"/\"openshift-service-ca.crt\"" Oct 30 19:43:25.199525 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.199519 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a7b7bc\"/\"default-dockercfg-zzgls\"" Oct 30 19:43:25.199624 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.199534 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a7b7bc\"/\"auth-test-model-a7b7bc-kserve-self-signed-certs\"" Oct 30 19:43:25.214382 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.214351 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s"] Oct 30 19:43:25.271480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.271440 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-home\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.271677 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.271491 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d13fe4-e24c-4831-80e9-d0994c89ee65-tls-certs\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.271677 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.271629 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-model-cache\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.271677 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.271656 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-dshm\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.271885 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.271864 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-tmp-dir\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.272038 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.272015 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtdt4\" (UniqueName: \"kubernetes.io/projected/67d13fe4-e24c-4831-80e9-d0994c89ee65-kube-api-access-dtdt4\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.373125 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.373078 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-tmp-dir\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.373353 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.373218 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtdt4\" (UniqueName: \"kubernetes.io/projected/67d13fe4-e24c-4831-80e9-d0994c89ee65-kube-api-access-dtdt4\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.373353 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.373321 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-home\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.373353 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.373344 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d13fe4-e24c-4831-80e9-d0994c89ee65-tls-certs\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.373531 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.373386 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-model-cache\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.373531 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.373408 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-dshm\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.373531 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.373514 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-tmp-dir\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.373757 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.373737 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-home\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.373806 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.373757 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-model-cache\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.375756 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.375730 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-dshm\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.375954 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.375935 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d13fe4-e24c-4831-80e9-d0994c89ee65-tls-certs\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.392092 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.392062 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtdt4\" (UniqueName: \"kubernetes.io/projected/67d13fe4-e24c-4831-80e9-d0994c89ee65-kube-api-access-dtdt4\") pod \"auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.506031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.502354 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:25.969830 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:43:25.969783 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67d13fe4_e24c_4831_80e9_d0994c89ee65.slice/crio-1514c5e2841d521ad811b6afaf8fa9f427ed428654191a09451201db1901e617 WatchSource:0}: Error finding container 1514c5e2841d521ad811b6afaf8fa9f427ed428654191a09451201db1901e617: Status 404 returned error can't find the container with id 1514c5e2841d521ad811b6afaf8fa9f427ed428654191a09451201db1901e617 Oct 30 19:43:25.973765 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:25.973732 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s"] Oct 30 19:43:26.069033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:26.068980 5047 patch_prober.go:29] interesting pod/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 19:43:26.069247 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:26.069070 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" podUID="2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 19:43:26.500818 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:26.500648 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" event={"ID":"67d13fe4-e24c-4831-80e9-d0994c89ee65","Type":"ContainerStarted","Data":"0e9a3461d74a244c21e19f21e39a363cd1cbf0e1f1fd4cdb4e7b8ec693852da6"} Oct 30 19:43:26.500818 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:26.500711 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" event={"ID":"67d13fe4-e24c-4831-80e9-d0994c89ee65","Type":"ContainerStarted","Data":"1514c5e2841d521ad811b6afaf8fa9f427ed428654191a09451201db1901e617"} Oct 30 19:43:26.503054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:26.503027 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:26.560041 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:26.559625 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" podStartSLOduration=1.559600608 podStartE2EDuration="1.559600608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:26.557350409 +0000 UTC m=+2359.631805769" watchObservedRunningTime="2026-10-30 19:43:26.559600608 +0000 UTC m=+2359.634055971" Oct 30 19:43:26.662682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:26.659129 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:43:27.068458 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:27.068421 5047 patch_prober.go:29] interesting pod/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 19:43:27.068760 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:27.068733 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" podUID="2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 19:43:28.070511 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:28.070464 5047 patch_prober.go:29] interesting pod/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 19:43:28.070952 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:28.070550 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" podUID="2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 19:43:29.068883 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:29.068845 5047 patch_prober.go:29] interesting pod/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 19:43:29.069091 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:29.068907 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" podUID="2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 19:43:30.069341 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:30.069300 5047 patch_prober.go:29] interesting pod/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 19:43:30.069858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:30.069371 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" podUID="2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 19:43:31.073506 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:31.073469 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:43:31.074855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:31.074832 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:43:35.503520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:35.503479 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:35.516452 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:35.516421 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:35.618233 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:35.618201 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:43:39.605367 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:43:39.605328 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:43:44.063855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.063816 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr"] Oct 30 19:43:44.069333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.069307 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 19:43:44.086133 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.086098 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-095f55-serving-cert\"" Oct 30 19:43:44.089890 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.089858 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr"] Oct 30 19:43:44.206328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.202001 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b63ca7-418b-44c0-a603-575c14f97efb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr\" (UID: \"d4b63ca7-418b-44c0-a603-575c14f97efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 19:43:44.206328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.205631 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82bx6\" (UniqueName: \"kubernetes.io/projected/d4b63ca7-418b-44c0-a603-575c14f97efb-kube-api-access-82bx6\") pod \"maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr\" (UID: \"d4b63ca7-418b-44c0-a603-575c14f97efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 19:43:44.206328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.205849 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b63ca7-418b-44c0-a603-575c14f97efb-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr\" (UID: \"d4b63ca7-418b-44c0-a603-575c14f97efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 19:43:44.206328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.205948 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b63ca7-418b-44c0-a603-575c14f97efb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr\" (UID: \"d4b63ca7-418b-44c0-a603-575c14f97efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 19:43:44.307991 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.307953 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b63ca7-418b-44c0-a603-575c14f97efb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr\" (UID: \"d4b63ca7-418b-44c0-a603-575c14f97efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 19:43:44.308227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.308086 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bx6\" (UniqueName: \"kubernetes.io/projected/d4b63ca7-418b-44c0-a603-575c14f97efb-kube-api-access-82bx6\") pod \"maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr\" (UID: \"d4b63ca7-418b-44c0-a603-575c14f97efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 19:43:44.308227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.308197 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b63ca7-418b-44c0-a603-575c14f97efb-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr\" (UID: \"d4b63ca7-418b-44c0-a603-575c14f97efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 19:43:44.308364 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.308259 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b63ca7-418b-44c0-a603-575c14f97efb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr\" (UID: \"d4b63ca7-418b-44c0-a603-575c14f97efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 19:43:44.309126 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.309087 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b63ca7-418b-44c0-a603-575c14f97efb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr\" (UID: \"d4b63ca7-418b-44c0-a603-575c14f97efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 19:43:44.311824 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.311798 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b63ca7-418b-44c0-a603-575c14f97efb-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr\" (UID: \"d4b63ca7-418b-44c0-a603-575c14f97efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 19:43:44.315216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.314757 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b63ca7-418b-44c0-a603-575c14f97efb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr\" (UID: \"d4b63ca7-418b-44c0-a603-575c14f97efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 19:43:44.326904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.326853 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82bx6\" (UniqueName: \"kubernetes.io/projected/d4b63ca7-418b-44c0-a603-575c14f97efb-kube-api-access-82bx6\") pod \"maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr\" (UID: \"d4b63ca7-418b-44c0-a603-575c14f97efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 19:43:44.388158 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.388116 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 19:43:44.697697 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.697667 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr"] Oct 30 19:43:44.697843 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:43:44.697816 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b63ca7_418b_44c0_a603_575c14f97efb.slice/crio-d87a4ccac66f5f77185b33b072405c8880c3ca3c7fdb2646816f1497deed404b WatchSource:0}: Error finding container d87a4ccac66f5f77185b33b072405c8880c3ca3c7fdb2646816f1497deed404b: Status 404 returned error can't find the container with id d87a4ccac66f5f77185b33b072405c8880c3ca3c7fdb2646816f1497deed404b Oct 30 19:43:44.732992 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:44.732951 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" event={"ID":"d4b63ca7-418b-44c0-a603-575c14f97efb","Type":"ContainerStarted","Data":"d87a4ccac66f5f77185b33b072405c8880c3ca3c7fdb2646816f1497deed404b"} Oct 30 19:43:45.745694 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:45.745652 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" event={"ID":"d4b63ca7-418b-44c0-a603-575c14f97efb","Type":"ContainerStarted","Data":"67a18d6f9c1680a5bdc11d17927a2c16cb514b48981ec131c6adc7cc0560e7de"} Oct 30 19:43:45.746263 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:45.745743 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 19:43:45.795598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:45.795529 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" podStartSLOduration=1.440911332 podStartE2EDuration="1.795512046s" totalImagesPullingTime="354.600714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:45.790847127 +0000 UTC m=+2378.865302487" watchObservedRunningTime="2026-10-30 19:43:45.795512046 +0000 UTC m=+2378.869967404" Oct 30 19:43:50.484483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.484277 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd"] Oct 30 19:43:50.491402 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.491366 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.507087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.507055 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd"] Oct 30 19:43:50.512055 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.510273 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-576a62-openshift-default-dockercfg-5l8zj\"" Oct 30 19:43:50.641269 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.641216 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-workload-socket\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.641504 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.641389 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfbzt\" (UniqueName: \"kubernetes.io/projected/df7012d4-5d6b-4586-a4ac-629154c1d347-kube-api-access-lfbzt\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.641504 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.641435 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7012d4-5d6b-4586-a4ac-629154c1d347-istiod-ca-cert\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.641666 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.641634 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-envoy\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.641731 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.641676 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-credential-socket\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.641731 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.641708 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-token\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.641835 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.641781 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-workload-certs\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.641896 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.641843 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-data\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.641963 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.641892 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-podinfo\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.755080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.751659 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbzt\" (UniqueName: \"kubernetes.io/projected/df7012d4-5d6b-4586-a4ac-629154c1d347-kube-api-access-lfbzt\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.755080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.752091 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7012d4-5d6b-4586-a4ac-629154c1d347-istiod-ca-cert\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.755080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.752274 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-envoy\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.755080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.752311 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-credential-socket\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.755080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.752341 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-token\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.755080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.752425 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-workload-certs\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.755080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.752491 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-data\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.755080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.752541 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-podinfo\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.755080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.752732 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-workload-socket\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.755080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.753289 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-workload-socket\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.755080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.753967 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7012d4-5d6b-4586-a4ac-629154c1d347-istiod-ca-cert\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.755080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.754508 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-workload-certs\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.755080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.754772 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-credential-socket\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.768765 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.768261 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-data\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.803848 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.803779 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-podinfo\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.834033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.828783 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfbzt\" (UniqueName: \"kubernetes.io/projected/df7012d4-5d6b-4586-a4ac-629154c1d347-kube-api-access-lfbzt\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.843803 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.843766 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-token\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:50.873052 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:50.873000 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-envoy\") pod \"e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:51.121383 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:51.121344 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:51.458491 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:51.458451 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd"] Oct 30 19:43:51.458651 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:43:51.458569 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7012d4_5d6b_4586_a4ac_629154c1d347.slice/crio-baad2ecbe9fdec62ad904c7e4cc8b305482dbe25937d9f42ce81e8b7540c48f3 WatchSource:0}: Error finding container baad2ecbe9fdec62ad904c7e4cc8b305482dbe25937d9f42ce81e8b7540c48f3: Status 404 returned error can't find the container with id baad2ecbe9fdec62ad904c7e4cc8b305482dbe25937d9f42ce81e8b7540c48f3 Oct 30 19:43:51.462036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:51.461982 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:43:51.462129 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:51.462080 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:43:51.462172 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:51.462129 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:43:51.779096 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:51.777206 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 19:43:51.873542 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:51.873503 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" event={"ID":"df7012d4-5d6b-4586-a4ac-629154c1d347","Type":"ContainerStarted","Data":"6bfa287be7a9eddd82860bbb2bb4a305fb81803b776a95b7c127f09a31a7f7f6"} Oct 30 19:43:51.873817 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:51.873796 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" event={"ID":"df7012d4-5d6b-4586-a4ac-629154c1d347","Type":"ContainerStarted","Data":"baad2ecbe9fdec62ad904c7e4cc8b305482dbe25937d9f42ce81e8b7540c48f3"} Oct 30 19:43:51.931576 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:51.931510 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" podStartSLOduration=1.931491517 podStartE2EDuration="1.931491517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:51.92678261 +0000 UTC m=+2385.001237981" watchObservedRunningTime="2026-10-30 19:43:51.931491517 +0000 UTC m=+2385.005946876" Oct 30 19:43:52.123031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:52.122657 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:52.136071 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:52.135128 5047 patch_prober.go:29] interesting pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 19:43:52.136071 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:52.135205 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" podUID="df7012d4-5d6b-4586-a4ac-629154c1d347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 19:43:53.146531 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:53.146496 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:53.961036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:53.958431 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:53.988052 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:53.987990 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:43:55.036034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:55.026357 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" podUID="a306b8ac-1021-4e38-be1c-aea68485bea3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:55.227032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:55.226452 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" podUID="3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.166:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:57.446812 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:57.446765 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r"] Oct 30 19:43:57.453456 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:57.453421 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" Oct 30 19:43:57.479479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:57.479442 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r"] Oct 30 19:43:57.571033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:57.554224 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-576a62-dockercfg-dz6s2\"" Oct 30 19:43:57.571033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:57.554365 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-576a62\"" Oct 30 19:43:57.766087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:57.761414 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90483a7b-6fe0-499d-abbf-7e0461ebd934-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r\" (UID: \"90483a7b-6fe0-499d-abbf-7e0461ebd934\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" Oct 30 19:43:57.766087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:57.761807 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q54f\" (UniqueName: \"kubernetes.io/projected/90483a7b-6fe0-499d-abbf-7e0461ebd934-kube-api-access-2q54f\") pod \"payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r\" (UID: \"90483a7b-6fe0-499d-abbf-7e0461ebd934\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" Oct 30 19:43:57.864281 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:57.864232 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q54f\" (UniqueName: \"kubernetes.io/projected/90483a7b-6fe0-499d-abbf-7e0461ebd934-kube-api-access-2q54f\") pod \"payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r\" (UID: \"90483a7b-6fe0-499d-abbf-7e0461ebd934\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" Oct 30 19:43:57.864476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:57.864418 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90483a7b-6fe0-499d-abbf-7e0461ebd934-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r\" (UID: \"90483a7b-6fe0-499d-abbf-7e0461ebd934\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" Oct 30 19:43:57.865290 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:57.865267 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90483a7b-6fe0-499d-abbf-7e0461ebd934-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r\" (UID: \"90483a7b-6fe0-499d-abbf-7e0461ebd934\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" Oct 30 19:43:57.881464 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:57.881429 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q54f\" (UniqueName: \"kubernetes.io/projected/90483a7b-6fe0-499d-abbf-7e0461ebd934-kube-api-access-2q54f\") pod \"payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r\" (UID: \"90483a7b-6fe0-499d-abbf-7e0461ebd934\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" Oct 30 19:43:58.076020 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:58.075909 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" Oct 30 19:43:58.385883 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:43:58.385849 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90483a7b_6fe0_499d_abbf_7e0461ebd934.slice/crio-a1b0d159b93bc6e36aca75656941ca11e1d466473e23c338a7451d7520d69fcd WatchSource:0}: Error finding container a1b0d159b93bc6e36aca75656941ca11e1d466473e23c338a7451d7520d69fcd: Status 404 returned error can't find the container with id a1b0d159b93bc6e36aca75656941ca11e1d466473e23c338a7451d7520d69fcd Oct 30 19:43:58.387762 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:58.387728 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r"] Oct 30 19:43:59.113697 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:59.113658 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" event={"ID":"90483a7b-6fe0-499d-abbf-7e0461ebd934","Type":"ContainerStarted","Data":"3384b4ba5193e56db2f13522c01a6b68cc10c0f43254cc70c6a9e2a493dc9126"} Oct 30 19:43:59.114132 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:59.113715 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" event={"ID":"90483a7b-6fe0-499d-abbf-7e0461ebd934","Type":"ContainerStarted","Data":"a1b0d159b93bc6e36aca75656941ca11e1d466473e23c338a7451d7520d69fcd"} Oct 30 19:43:59.114132 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:59.113885 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" Oct 30 19:43:59.248040 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:43:59.234509 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" podStartSLOduration=2.23448738 podStartE2EDuration="2.23448738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:59.162315546 +0000 UTC m=+2392.236770905" watchObservedRunningTime="2026-10-30 19:43:59.23448738 +0000 UTC m=+2392.308942740" Oct 30 19:44:07.796001 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:07.795966 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:44:07.797041 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:07.796979 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:44:09.603630 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:44:09.603591 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:44:10.128335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:10.128306 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" Oct 30 19:44:29.307803 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:29.307761 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5"] Oct 30 19:44:33.938308 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:33.938269 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c"] Oct 30 19:44:33.938750 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:33.938509 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" podUID="8f5a1117-41f6-421e-a9cc-9230eea118f8" containerName="istio-proxy" containerID="cri-o://0d1a88c3b7420c62a8b16df929c0c158f0354070793b1e28b579ad70ca58dc64" gracePeriod=30 Oct 30 19:44:33.975019 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:33.974961 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:44:33.975322 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:33.975275 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" podUID="ea906da9-9a7c-4755-b5f1-2f6388875be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:35.563109 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:35.563070 5047 generic.go:356] "Generic (PLEG): container finished" podID="ea906da9-9a7c-4755-b5f1-2f6388875be2" containerID="004460907acd6a0cfa6d797c7623ed7e37abdd83f0532b4b4862c59d34dbd984" exitCode=0 Oct 30 19:44:35.563562 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:35.563142 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" event={"ID":"ea906da9-9a7c-4755-b5f1-2f6388875be2","Type":"ContainerDied","Data":"004460907acd6a0cfa6d797c7623ed7e37abdd83f0532b4b4862c59d34dbd984"} Oct 30 19:44:35.563723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:35.563704 5047 scope.go:122] "RemoveContainer" containerID="004460907acd6a0cfa6d797c7623ed7e37abdd83f0532b4b4862c59d34dbd984" Oct 30 19:44:35.565920 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:35.565882 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:44:35.566043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:35.565965 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:44:35.566043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:35.566000 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:44:36.574707 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:36.574667 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" event={"ID":"ea906da9-9a7c-4755-b5f1-2f6388875be2","Type":"ContainerStarted","Data":"2c6a99184a1d0a2bc0227341117cc83a4c2084c1dbe547a6ae90ee730309920e"} Oct 30 19:44:37.119573 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:37.119533 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:44:37.119831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:37.119804 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 19:44:37.119879 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:37.119859 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" podUID="ea906da9-9a7c-4755-b5f1-2f6388875be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 19:44:38.119176 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:38.119140 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 19:44:38.119584 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:38.119195 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" podUID="ea906da9-9a7c-4755-b5f1-2f6388875be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 19:44:38.460707 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:38.460601 5047 patch_prober.go:29] interesting pod/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:44:38.460982 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:38.460950 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" podUID="8f5a1117-41f6-421e-a9cc-9230eea118f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:44:39.119895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.119591 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 19:44:39.119895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.119646 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" podUID="ea906da9-9a7c-4755-b5f1-2f6388875be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 19:44:39.213626 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.213596 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:44:39.313078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.312965 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-token\" (UniqueName: \"kubernetes.io/projected/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-token\") pod \"8f5a1117-41f6-421e-a9cc-9230eea118f8\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " Oct 30 19:44:39.313507 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.313469 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-data\") pod \"8f5a1117-41f6-421e-a9cc-9230eea118f8\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " Oct 30 19:44:39.313642 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.313565 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5a1117-41f6-421e-a9cc-9230eea118f8-kube-api-access-9zvkb\" (UniqueName: \"kubernetes.io/projected/8f5a1117-41f6-421e-a9cc-9230eea118f8-kube-api-access-9zvkb\") pod \"8f5a1117-41f6-421e-a9cc-9230eea118f8\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " Oct 30 19:44:39.313642 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.313603 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-workload-certs\") pod \"8f5a1117-41f6-421e-a9cc-9230eea118f8\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " Oct 30 19:44:39.313780 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.313643 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f5a1117-41f6-421e-a9cc-9230eea118f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5a1117-41f6-421e-a9cc-9230eea118f8-istiod-ca-cert\") pod \"8f5a1117-41f6-421e-a9cc-9230eea118f8\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " Oct 30 19:44:39.313780 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.313684 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-envoy\") pod \"8f5a1117-41f6-421e-a9cc-9230eea118f8\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " Oct 30 19:44:39.313898 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.313812 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-credential-socket\") pod \"8f5a1117-41f6-421e-a9cc-9230eea118f8\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " Oct 30 19:44:39.313898 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.313850 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-podinfo\") pod \"8f5a1117-41f6-421e-a9cc-9230eea118f8\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " Oct 30 19:44:39.314056 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.313942 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-workload-socket\") pod \"8f5a1117-41f6-421e-a9cc-9230eea118f8\" (UID: \"8f5a1117-41f6-421e-a9cc-9230eea118f8\") " Oct 30 19:44:39.314444 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.314417 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f5a1117-41f6-421e-a9cc-9230eea118f8-istiod-ca-cert" pod "8f5a1117-41f6-421e-a9cc-9230eea118f8" (UID: "8f5a1117-41f6-421e-a9cc-9230eea118f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:39.314726 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.314699 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-workload-socket" pod "8f5a1117-41f6-421e-a9cc-9230eea118f8" (UID: "8f5a1117-41f6-421e-a9cc-9230eea118f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:39.314879 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.314860 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-data" pod "8f5a1117-41f6-421e-a9cc-9230eea118f8" (UID: "8f5a1117-41f6-421e-a9cc-9230eea118f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:39.314956 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.314901 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5a1117-41f6-421e-a9cc-9230eea118f8-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:39.314956 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.314921 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:39.315272 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.315247 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-credential-socket" pod "8f5a1117-41f6-421e-a9cc-9230eea118f8" (UID: "8f5a1117-41f6-421e-a9cc-9230eea118f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:39.315817 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.315785 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-token" pod "8f5a1117-41f6-421e-a9cc-9230eea118f8" (UID: "8f5a1117-41f6-421e-a9cc-9230eea118f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:39.316129 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.316102 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-workload-certs" pod "8f5a1117-41f6-421e-a9cc-9230eea118f8" (UID: "8f5a1117-41f6-421e-a9cc-9230eea118f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:39.317258 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.317225 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5a1117-41f6-421e-a9cc-9230eea118f8-kube-api-access-9zvkb" pod "8f5a1117-41f6-421e-a9cc-9230eea118f8" (UID: "8f5a1117-41f6-421e-a9cc-9230eea118f8"). InnerVolumeSpecName "kube-api-access-9zvkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:39.317558 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.317529 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-envoy" pod "8f5a1117-41f6-421e-a9cc-9230eea118f8" (UID: "8f5a1117-41f6-421e-a9cc-9230eea118f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:39.318193 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.318170 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-podinfo" pod "8f5a1117-41f6-421e-a9cc-9230eea118f8" (UID: "8f5a1117-41f6-421e-a9cc-9230eea118f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:44:39.416822 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.416788 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:39.416822 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.416818 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:39.416822 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.416829 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zvkb\" (UniqueName: \"kubernetes.io/projected/8f5a1117-41f6-421e-a9cc-9230eea118f8-kube-api-access-9zvkb\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:39.417095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.416838 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:39.417095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.416847 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:39.417095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.416855 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5a1117-41f6-421e-a9cc-9230eea118f8-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:39.417095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.416864 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5a1117-41f6-421e-a9cc-9230eea118f8-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:39.603451 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.603412 5047 generic.go:356] "Generic (PLEG): container finished" podID="8f5a1117-41f6-421e-a9cc-9230eea118f8" containerID="0d1a88c3b7420c62a8b16df929c0c158f0354070793b1e28b579ad70ca58dc64" exitCode=0 Oct 30 19:44:39.603643 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.603480 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" Oct 30 19:44:39.603643 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.603493 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" event={"ID":"8f5a1117-41f6-421e-a9cc-9230eea118f8","Type":"ContainerDied","Data":"0d1a88c3b7420c62a8b16df929c0c158f0354070793b1e28b579ad70ca58dc64"} Oct 30 19:44:39.603643 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.603549 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c" event={"ID":"8f5a1117-41f6-421e-a9cc-9230eea118f8","Type":"ContainerDied","Data":"e12531ba5fde50cd0efb1c500f70382a1f47b5613e063b61d4ef2eb2a167e75f"} Oct 30 19:44:39.603643 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.603572 5047 scope.go:122] "RemoveContainer" containerID="0d1a88c3b7420c62a8b16df929c0c158f0354070793b1e28b579ad70ca58dc64" Oct 30 19:44:39.614033 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:44:39.612160 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:44:39.621500 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.621404 5047 scope.go:122] "RemoveContainer" containerID="0d1a88c3b7420c62a8b16df929c0c158f0354070793b1e28b579ad70ca58dc64" Oct 30 19:44:39.621799 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:44:39.621747 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1a88c3b7420c62a8b16df929c0c158f0354070793b1e28b579ad70ca58dc64\": container with ID starting with 0d1a88c3b7420c62a8b16df929c0c158f0354070793b1e28b579ad70ca58dc64 not found: ID does not exist" containerID="0d1a88c3b7420c62a8b16df929c0c158f0354070793b1e28b579ad70ca58dc64" Oct 30 19:44:39.621799 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.621775 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1a88c3b7420c62a8b16df929c0c158f0354070793b1e28b579ad70ca58dc64"} err="failed to get container status \"0d1a88c3b7420c62a8b16df929c0c158f0354070793b1e28b579ad70ca58dc64\": rpc error: code = NotFound desc = could not find container \"0d1a88c3b7420c62a8b16df929c0c158f0354070793b1e28b579ad70ca58dc64\": container with ID starting with 0d1a88c3b7420c62a8b16df929c0c158f0354070793b1e28b579ad70ca58dc64 not found: ID does not exist" Oct 30 19:44:39.700882 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.700850 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c"] Oct 30 19:44:39.708548 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.708491 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-w6h9c"] Oct 30 19:44:39.903082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.902986 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft"] Oct 30 19:44:39.904184 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.904163 5047 cpu_manager.go:414] "containerMap: removing container" podUID="8f5a1117-41f6-421e-a9cc-9230eea118f8" containerName="istio-proxy" Oct 30 19:44:39.904306 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.904296 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5a1117-41f6-421e-a9cc-9230eea118f8" containerName="istio-proxy" Oct 30 19:44:39.904584 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.904572 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5a1117-41f6-421e-a9cc-9230eea118f8" containerName="istio-proxy" Oct 30 19:44:39.908790 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.908772 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:44:39.923898 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.923866 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/980112bf-3559-4a54-aea8-4ded5e0f269e-maas-api-tls\") pod \"maas-api-e2e-inf-b-576a62-55f5785447-j2vft\" (UID: \"980112bf-3559-4a54-aea8-4ded5e0f269e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:44:39.924028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.923989 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980112bf-3559-4a54-aea8-4ded5e0f269e-metrics-certs\") pod \"maas-api-e2e-inf-b-576a62-55f5785447-j2vft\" (UID: \"980112bf-3559-4a54-aea8-4ded5e0f269e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:44:39.924166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.924146 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/980112bf-3559-4a54-aea8-4ded5e0f269e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-576a62-55f5785447-j2vft\" (UID: \"980112bf-3559-4a54-aea8-4ded5e0f269e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:44:39.924207 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.924193 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm9n8\" (UniqueName: \"kubernetes.io/projected/980112bf-3559-4a54-aea8-4ded5e0f269e-kube-api-access-xm9n8\") pod \"maas-api-e2e-inf-b-576a62-55f5785447-j2vft\" (UID: \"980112bf-3559-4a54-aea8-4ded5e0f269e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:44:39.929282 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.929256 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-576a62-serving-cert\"" Oct 30 19:44:39.982141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:39.978219 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft"] Oct 30 19:44:40.028951 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:40.028905 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/980112bf-3559-4a54-aea8-4ded5e0f269e-maas-api-tls\") pod \"maas-api-e2e-inf-b-576a62-55f5785447-j2vft\" (UID: \"980112bf-3559-4a54-aea8-4ded5e0f269e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:44:40.029145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:40.029060 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980112bf-3559-4a54-aea8-4ded5e0f269e-metrics-certs\") pod \"maas-api-e2e-inf-b-576a62-55f5785447-j2vft\" (UID: \"980112bf-3559-4a54-aea8-4ded5e0f269e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:44:40.029223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:40.029208 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/980112bf-3559-4a54-aea8-4ded5e0f269e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-576a62-55f5785447-j2vft\" (UID: \"980112bf-3559-4a54-aea8-4ded5e0f269e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:44:40.029281 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:40.029264 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm9n8\" (UniqueName: \"kubernetes.io/projected/980112bf-3559-4a54-aea8-4ded5e0f269e-kube-api-access-xm9n8\") pod \"maas-api-e2e-inf-b-576a62-55f5785447-j2vft\" (UID: \"980112bf-3559-4a54-aea8-4ded5e0f269e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:44:40.029965 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:44:40.029776 5047 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-serving-cert: secret "maas-api-e2e-inf-b-576a62-serving-cert" not found Oct 30 19:44:40.029965 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:44:40.029856 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/980112bf-3559-4a54-aea8-4ded5e0f269e-maas-api-tls podName:980112bf-3559-4a54-aea8-4ded5e0f269e nodeName:}" failed. No retries permitted until 2026-10-30 19:44:40.529834733 +0000 UTC m=+2433.604290090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/980112bf-3559-4a54-aea8-4ded5e0f269e-maas-api-tls") pod "maas-api-e2e-inf-b-576a62-55f5785447-j2vft" (UID: "980112bf-3559-4a54-aea8-4ded5e0f269e") : secret "maas-api-e2e-inf-b-576a62-serving-cert" not found Oct 30 19:44:40.031831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:40.031808 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/980112bf-3559-4a54-aea8-4ded5e0f269e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-576a62-55f5785447-j2vft\" (UID: \"980112bf-3559-4a54-aea8-4ded5e0f269e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:44:40.075163 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:40.072957 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980112bf-3559-4a54-aea8-4ded5e0f269e-metrics-certs\") pod \"maas-api-e2e-inf-b-576a62-55f5785447-j2vft\" (UID: \"980112bf-3559-4a54-aea8-4ded5e0f269e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:44:40.078175 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:40.078147 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm9n8\" (UniqueName: \"kubernetes.io/projected/980112bf-3559-4a54-aea8-4ded5e0f269e-kube-api-access-xm9n8\") pod \"maas-api-e2e-inf-b-576a62-55f5785447-j2vft\" (UID: \"980112bf-3559-4a54-aea8-4ded5e0f269e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:44:40.119272 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:40.119239 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 19:44:40.119508 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:40.119485 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" podUID="ea906da9-9a7c-4755-b5f1-2f6388875be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 19:44:40.560532 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:40.560493 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/980112bf-3559-4a54-aea8-4ded5e0f269e-maas-api-tls\") pod \"maas-api-e2e-inf-b-576a62-55f5785447-j2vft\" (UID: \"980112bf-3559-4a54-aea8-4ded5e0f269e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:44:40.564140 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:40.564108 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/980112bf-3559-4a54-aea8-4ded5e0f269e-maas-api-tls\") pod \"maas-api-e2e-inf-b-576a62-55f5785447-j2vft\" (UID: \"980112bf-3559-4a54-aea8-4ded5e0f269e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:44:40.824983 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:40.824896 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:44:41.088744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:41.088695 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f5a1117-41f6-421e-a9cc-9230eea118f8" path="/var/lib/kubelet/pods/8f5a1117-41f6-421e-a9cc-9230eea118f8/volumes" Oct 30 19:44:41.118970 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:41.118724 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 19:44:41.118970 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:41.118777 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" podUID="ea906da9-9a7c-4755-b5f1-2f6388875be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 19:44:41.232965 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:41.232080 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft"] Oct 30 19:44:41.257028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:41.234304 5047 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:44:41.623311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:41.623271 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" event={"ID":"980112bf-3559-4a54-aea8-4ded5e0f269e","Type":"ContainerStarted","Data":"e8c46c9c15b4cf6e4836e4c1e47e266c7f8dfe4984f07d6dbeada5fc15bb13c5"} Oct 30 19:44:42.123838 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:42.123807 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:44:42.124283 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:42.124256 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:44:42.124939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:42.124917 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 19:44:42.633297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:42.633260 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" event={"ID":"980112bf-3559-4a54-aea8-4ded5e0f269e","Type":"ContainerStarted","Data":"27d511ae0bcd590d90fb59331c6d8433029f06bfd8834dc50ec23fc00cf5c7ef"} Oct 30 19:44:42.633715 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:42.633349 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:44:42.683311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:42.683259 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" podStartSLOduration=3.310344795 podStartE2EDuration="3.683244582s" totalImagesPullingTime="372.899787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:42.682269508 +0000 UTC m=+2435.756724867" watchObservedRunningTime="2026-10-30 19:44:42.683244582 +0000 UTC m=+2435.757699974" Oct 30 19:44:44.335334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.335265 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" podUID="71d8d155-8327-4233-9fae-0d11e447bb0f" containerName="main" containerID="cri-o://74c2657dc3c8dbb7c25b5687a1e6c10a4b068907298d71d1ff6cad5ec9b200f6" gracePeriod=45 Oct 30 19:44:44.450542 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.450468 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22"] Oct 30 19:44:44.747712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.747387 5047 generic.go:356] "Generic (PLEG): container finished" podID="71d8d155-8327-4233-9fae-0d11e447bb0f" containerID="74c2657dc3c8dbb7c25b5687a1e6c10a4b068907298d71d1ff6cad5ec9b200f6" exitCode=0 Oct 30 19:44:44.747712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.747606 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" event={"ID":"71d8d155-8327-4233-9fae-0d11e447bb0f","Type":"ContainerDied","Data":"74c2657dc3c8dbb7c25b5687a1e6c10a4b068907298d71d1ff6cad5ec9b200f6"} Oct 30 19:44:44.827378 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.827351 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:44:44.932897 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.932858 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-model-cache\") pod \"71d8d155-8327-4233-9fae-0d11e447bb0f\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " Oct 30 19:44:44.933144 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.932972 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-dshm\") pod \"71d8d155-8327-4233-9fae-0d11e447bb0f\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " Oct 30 19:44:44.933144 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.933114 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71d8d155-8327-4233-9fae-0d11e447bb0f-tls-certs\" (UniqueName: \"kubernetes.io/secret/71d8d155-8327-4233-9fae-0d11e447bb0f-tls-certs\") pod \"71d8d155-8327-4233-9fae-0d11e447bb0f\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " Oct 30 19:44:44.933284 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.933193 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-model-cache" pod "71d8d155-8327-4233-9fae-0d11e447bb0f" (UID: "71d8d155-8327-4233-9fae-0d11e447bb0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:44.933284 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.933207 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d8d155-8327-4233-9fae-0d11e447bb0f-kube-api-access-xl5zj\" (UniqueName: \"kubernetes.io/projected/71d8d155-8327-4233-9fae-0d11e447bb0f-kube-api-access-xl5zj\") pod \"71d8d155-8327-4233-9fae-0d11e447bb0f\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " Oct 30 19:44:44.933284 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.933262 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-tmp-dir\") pod \"71d8d155-8327-4233-9fae-0d11e447bb0f\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " Oct 30 19:44:44.933441 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.933326 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-home\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-home\") pod \"71d8d155-8327-4233-9fae-0d11e447bb0f\" (UID: \"71d8d155-8327-4233-9fae-0d11e447bb0f\") " Oct 30 19:44:44.933621 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.933596 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-tmp-dir" pod "71d8d155-8327-4233-9fae-0d11e447bb0f" (UID: "71d8d155-8327-4233-9fae-0d11e447bb0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:44.933762 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.933630 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-home" pod "71d8d155-8327-4233-9fae-0d11e447bb0f" (UID: "71d8d155-8327-4233-9fae-0d11e447bb0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:44.934076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.934057 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:44.934166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.934080 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:44.934166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.934093 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:44.935544 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.935523 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d8d155-8327-4233-9fae-0d11e447bb0f-tls-certs" pod "71d8d155-8327-4233-9fae-0d11e447bb0f" (UID: "71d8d155-8327-4233-9fae-0d11e447bb0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:44.935630 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.935523 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d8d155-8327-4233-9fae-0d11e447bb0f-kube-api-access-xl5zj" pod "71d8d155-8327-4233-9fae-0d11e447bb0f" (UID: "71d8d155-8327-4233-9fae-0d11e447bb0f"). InnerVolumeSpecName "kube-api-access-xl5zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:44.935841 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:44.935824 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-dshm" pod "71d8d155-8327-4233-9fae-0d11e447bb0f" (UID: "71d8d155-8327-4233-9fae-0d11e447bb0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:45.035819 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:45.035734 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl5zj\" (UniqueName: \"kubernetes.io/projected/71d8d155-8327-4233-9fae-0d11e447bb0f-kube-api-access-xl5zj\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:45.035819 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:45.035779 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d8d155-8327-4233-9fae-0d11e447bb0f-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:45.035819 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:45.035796 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d8d155-8327-4233-9fae-0d11e447bb0f-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:45.725449 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:45.725398 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" podUID="71d8d155-8327-4233-9fae-0d11e447bb0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.157:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:45.757575 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:45.757545 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" Oct 30 19:44:45.757773 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:45.757586 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5" event={"ID":"71d8d155-8327-4233-9fae-0d11e447bb0f","Type":"ContainerDied","Data":"028c5fe5647ef93802690ab53f73d7125f100d55951f5917a9506e7d8c4efdc6"} Oct 30 19:44:45.757773 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:45.757639 5047 scope.go:122] "RemoveContainer" containerID="74c2657dc3c8dbb7c25b5687a1e6c10a4b068907298d71d1ff6cad5ec9b200f6" Oct 30 19:44:45.810161 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:45.810129 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5"] Oct 30 19:44:45.819364 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:45.819332 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5d92fa/test-model-5d92fa-kserve-74579d55b9-tnzj5"] Oct 30 19:44:46.543505 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:46.543469 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t"] Oct 30 19:44:46.544474 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:46.544450 5047 cpu_manager.go:414] "containerMap: removing container" podUID="71d8d155-8327-4233-9fae-0d11e447bb0f" containerName="main" Oct 30 19:44:46.544474 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:46.544473 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d8d155-8327-4233-9fae-0d11e447bb0f" containerName="main" Oct 30 19:44:46.544693 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:46.544677 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d8d155-8327-4233-9fae-0d11e447bb0f" containerName="main" Oct 30 19:44:46.550550 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:46.550526 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:46.564588 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:46.564541 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-870193-openshift-default-dockercfg-wpz2h\"" Oct 30 19:44:46.580669 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:46.580630 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t"] Oct 30 19:44:46.798080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:46.783119 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-workload-certs\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:46.882720 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:46.882685 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-credential-socket\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:46.882999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:46.882974 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05dd723-2d08-4261-aa55-e642430737e6-istio-token\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:46.890026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:46.883210 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn7vk\" (UniqueName: \"kubernetes.io/projected/a05dd723-2d08-4261-aa55-e642430737e6-kube-api-access-dn7vk\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:46.890307 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:46.890282 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-workload-socket\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:46.890434 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:46.890407 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05dd723-2d08-4261-aa55-e642430737e6-istiod-ca-cert\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:46.891430 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:46.890702 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-istio-envoy\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:46.906030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:46.891738 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05dd723-2d08-4261-aa55-e642430737e6-istio-podinfo\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:46.907399 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:46.907365 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-istio-data\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.009922 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.009886 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-istio-envoy\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.010097 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.009976 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05dd723-2d08-4261-aa55-e642430737e6-istio-podinfo\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.010097 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.010044 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-istio-data\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.010186 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.010121 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-workload-certs\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.010227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.010189 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-credential-socket\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.010260 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.010231 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05dd723-2d08-4261-aa55-e642430737e6-istio-token\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.010334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.010313 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn7vk\" (UniqueName: \"kubernetes.io/projected/a05dd723-2d08-4261-aa55-e642430737e6-kube-api-access-dn7vk\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.010396 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.010365 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-workload-socket\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.010396 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.010391 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05dd723-2d08-4261-aa55-e642430737e6-istiod-ca-cert\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.010527 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.010500 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-istio-data\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.010640 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.010622 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-credential-socket\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.010918 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.010896 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-workload-socket\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.011223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.011197 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-workload-certs\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.011278 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.011199 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05dd723-2d08-4261-aa55-e642430737e6-istiod-ca-cert\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.012905 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.012881 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-istio-envoy\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.013318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.013301 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05dd723-2d08-4261-aa55-e642430737e6-istio-podinfo\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.028415 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.028368 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05dd723-2d08-4261-aa55-e642430737e6-istio-token\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.028649 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.028628 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn7vk\" (UniqueName: \"kubernetes.io/projected/a05dd723-2d08-4261-aa55-e642430737e6-kube-api-access-dn7vk\") pod \"e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.056250 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.056180 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d8d155-8327-4233-9fae-0d11e447bb0f" path="/var/lib/kubelet/pods/71d8d155-8327-4233-9fae-0d11e447bb0f/volumes" Oct 30 19:44:47.179457 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.179420 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:47.516598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.516556 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t"] Oct 30 19:44:47.517035 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:44:47.516991 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05dd723_2d08_4261_aa55_e642430737e6.slice/crio-832eb5e97f58ed84312a98cff121694fab81f71310da7ecf0f7ec40bcfe28b5e WatchSource:0}: Error finding container 832eb5e97f58ed84312a98cff121694fab81f71310da7ecf0f7ec40bcfe28b5e: Status 404 returned error can't find the container with id 832eb5e97f58ed84312a98cff121694fab81f71310da7ecf0f7ec40bcfe28b5e Oct 30 19:44:47.519997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.519958 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:44:47.520131 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.520051 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:44:47.520131 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.520085 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:44:47.933902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.933860 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" event={"ID":"a05dd723-2d08-4261-aa55-e642430737e6","Type":"ContainerStarted","Data":"854d2769d396b31c54c5370335d46309f17672d4c1b0dbd59b793ab0e5c754a9"} Oct 30 19:44:47.934516 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:47.933962 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" event={"ID":"a05dd723-2d08-4261-aa55-e642430737e6","Type":"ContainerStarted","Data":"832eb5e97f58ed84312a98cff121694fab81f71310da7ecf0f7ec40bcfe28b5e"} Oct 30 19:44:48.006184 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:48.006122 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" podStartSLOduration=2.006104074 podStartE2EDuration="2.006104074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:47.992779881 +0000 UTC m=+2441.067235240" watchObservedRunningTime="2026-10-30 19:44:48.006104074 +0000 UTC m=+2441.080559434" Oct 30 19:44:48.183671 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:48.180379 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:48.213111 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:48.210478 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:48.646673 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:48.646636 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:44:48.941973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:48.941886 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:48.943137 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:48.943117 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:44:59.473757 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:59.473704 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" podUID="a306b8ac-1021-4e38-be1c-aea68485bea3" containerName="main" containerID="cri-o://2af3c1e898d95b3fccd2c376f6e036d5b5eac281e721bf49d21a16e4829a359a" gracePeriod=45 Oct 30 19:44:59.822419 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:59.822331 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:44:59.914504 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:59.914460 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a306b8ac-1021-4e38-be1c-aea68485bea3-tls-certs\" (UniqueName: \"kubernetes.io/secret/a306b8ac-1021-4e38-be1c-aea68485bea3-tls-certs\") pod \"a306b8ac-1021-4e38-be1c-aea68485bea3\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " Oct 30 19:44:59.914685 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:59.914561 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-home\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-home\") pod \"a306b8ac-1021-4e38-be1c-aea68485bea3\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " Oct 30 19:44:59.914685 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:59.914618 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-model-cache\") pod \"a306b8ac-1021-4e38-be1c-aea68485bea3\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " Oct 30 19:44:59.914685 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:59.914651 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-dshm\") pod \"a306b8ac-1021-4e38-be1c-aea68485bea3\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " Oct 30 19:44:59.914882 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:59.914835 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-tmp-dir\") pod \"a306b8ac-1021-4e38-be1c-aea68485bea3\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " Oct 30 19:44:59.914882 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:59.914866 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-home" pod "a306b8ac-1021-4e38-be1c-aea68485bea3" (UID: "a306b8ac-1021-4e38-be1c-aea68485bea3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:59.915027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:59.914984 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-model-cache" pod "a306b8ac-1021-4e38-be1c-aea68485bea3" (UID: "a306b8ac-1021-4e38-be1c-aea68485bea3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:59.915178 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:59.915157 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a306b8ac-1021-4e38-be1c-aea68485bea3-kube-api-access-d9zzg\" (UniqueName: \"kubernetes.io/projected/a306b8ac-1021-4e38-be1c-aea68485bea3-kube-api-access-d9zzg\") pod \"a306b8ac-1021-4e38-be1c-aea68485bea3\" (UID: \"a306b8ac-1021-4e38-be1c-aea68485bea3\") " Oct 30 19:44:59.915256 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:59.915230 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-tmp-dir" pod "a306b8ac-1021-4e38-be1c-aea68485bea3" (UID: "a306b8ac-1021-4e38-be1c-aea68485bea3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:59.915968 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:59.915933 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:59.915968 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:59.915962 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:59.916208 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:59.915974 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:44:59.918052 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:59.917911 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a306b8ac-1021-4e38-be1c-aea68485bea3-kube-api-access-d9zzg" pod "a306b8ac-1021-4e38-be1c-aea68485bea3" (UID: "a306b8ac-1021-4e38-be1c-aea68485bea3"). InnerVolumeSpecName "kube-api-access-d9zzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:59.918163 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:59.918048 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-dshm" pod "a306b8ac-1021-4e38-be1c-aea68485bea3" (UID: "a306b8ac-1021-4e38-be1c-aea68485bea3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:59.918253 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:44:59.918231 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a306b8ac-1021-4e38-be1c-aea68485bea3-tls-certs" pod "a306b8ac-1021-4e38-be1c-aea68485bea3" (UID: "a306b8ac-1021-4e38-be1c-aea68485bea3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:00.017408 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.017320 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a306b8ac-1021-4e38-be1c-aea68485bea3-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:00.017408 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.017353 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a306b8ac-1021-4e38-be1c-aea68485bea3-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:00.017408 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.017363 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9zzg\" (UniqueName: \"kubernetes.io/projected/a306b8ac-1021-4e38-be1c-aea68485bea3-kube-api-access-d9zzg\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:00.028139 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.028106 5047 generic.go:356] "Generic (PLEG): container finished" podID="a306b8ac-1021-4e38-be1c-aea68485bea3" containerID="2af3c1e898d95b3fccd2c376f6e036d5b5eac281e721bf49d21a16e4829a359a" exitCode=0 Oct 30 19:45:00.028319 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.028152 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" Oct 30 19:45:00.028319 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.028187 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" event={"ID":"a306b8ac-1021-4e38-be1c-aea68485bea3","Type":"ContainerDied","Data":"2af3c1e898d95b3fccd2c376f6e036d5b5eac281e721bf49d21a16e4829a359a"} Oct 30 19:45:00.028319 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.028233 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22" event={"ID":"a306b8ac-1021-4e38-be1c-aea68485bea3","Type":"ContainerDied","Data":"904d71f22c0d965922070ae9e9a839ec7973de6d997aa6a5c2e678993df13dc5"} Oct 30 19:45:00.028319 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.028257 5047 scope.go:122] "RemoveContainer" containerID="2af3c1e898d95b3fccd2c376f6e036d5b5eac281e721bf49d21a16e4829a359a" Oct 30 19:45:00.042227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.042204 5047 scope.go:122] "RemoveContainer" containerID="2af3c1e898d95b3fccd2c376f6e036d5b5eac281e721bf49d21a16e4829a359a" Oct 30 19:45:00.042550 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:00.042529 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af3c1e898d95b3fccd2c376f6e036d5b5eac281e721bf49d21a16e4829a359a\": container with ID starting with 2af3c1e898d95b3fccd2c376f6e036d5b5eac281e721bf49d21a16e4829a359a not found: ID does not exist" containerID="2af3c1e898d95b3fccd2c376f6e036d5b5eac281e721bf49d21a16e4829a359a" Oct 30 19:45:00.042611 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.042561 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af3c1e898d95b3fccd2c376f6e036d5b5eac281e721bf49d21a16e4829a359a"} err="failed to get container status \"2af3c1e898d95b3fccd2c376f6e036d5b5eac281e721bf49d21a16e4829a359a\": rpc error: code = NotFound desc = could not find container \"2af3c1e898d95b3fccd2c376f6e036d5b5eac281e721bf49d21a16e4829a359a\": container with ID starting with 2af3c1e898d95b3fccd2c376f6e036d5b5eac281e721bf49d21a16e4829a359a not found: ID does not exist" Oct 30 19:45:00.081385 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.081344 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22"] Oct 30 19:45:00.085524 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.085493 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-469724/test-model-469724-kserve-d45bd94f9-2sc22"] Oct 30 19:45:00.359758 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.359723 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls"] Oct 30 19:45:00.361143 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.361108 5047 cpu_manager.go:414] "containerMap: removing container" podUID="a306b8ac-1021-4e38-be1c-aea68485bea3" containerName="main" Oct 30 19:45:00.361143 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.361135 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a306b8ac-1021-4e38-be1c-aea68485bea3" containerName="main" Oct 30 19:45:00.361389 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.361375 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="a306b8ac-1021-4e38-be1c-aea68485bea3" containerName="main" Oct 30 19:45:00.367879 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.367850 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx"] Oct 30 19:45:00.368057 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.367970 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" Oct 30 19:45:00.374276 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.374249 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" Oct 30 19:45:00.405906 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.405874 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls"] Oct 30 19:45:00.412819 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.412784 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx"] Oct 30 19:45:00.422283 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.422250 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 19:45:00.525596 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.525561 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5csr\" (UniqueName: \"kubernetes.io/projected/4d140123-d166-4f06-8d0c-ed0bb574f0e1-kube-api-access-z5csr\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls\" (UID: \"4d140123-d166-4f06-8d0c-ed0bb574f0e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" Oct 30 19:45:00.526202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.525621 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc67df7-4435-4033-95dd-c729c7357881-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx\" (UID: \"adc67df7-4435-4033-95dd-c729c7357881\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" Oct 30 19:45:00.526202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.525756 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d140123-d166-4f06-8d0c-ed0bb574f0e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls\" (UID: \"4d140123-d166-4f06-8d0c-ed0bb574f0e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" Oct 30 19:45:00.526202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.525907 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfbt4\" (UniqueName: \"kubernetes.io/projected/adc67df7-4435-4033-95dd-c729c7357881-kube-api-access-vfbt4\") pod \"maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx\" (UID: \"adc67df7-4435-4033-95dd-c729c7357881\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" Oct 30 19:45:00.627379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.627287 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5csr\" (UniqueName: \"kubernetes.io/projected/4d140123-d166-4f06-8d0c-ed0bb574f0e1-kube-api-access-z5csr\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls\" (UID: \"4d140123-d166-4f06-8d0c-ed0bb574f0e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" Oct 30 19:45:00.627379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.627348 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc67df7-4435-4033-95dd-c729c7357881-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx\" (UID: \"adc67df7-4435-4033-95dd-c729c7357881\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" Oct 30 19:45:00.627573 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.627384 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d140123-d166-4f06-8d0c-ed0bb574f0e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls\" (UID: \"4d140123-d166-4f06-8d0c-ed0bb574f0e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" Oct 30 19:45:00.627573 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.627471 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfbt4\" (UniqueName: \"kubernetes.io/projected/adc67df7-4435-4033-95dd-c729c7357881-kube-api-access-vfbt4\") pod \"maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx\" (UID: \"adc67df7-4435-4033-95dd-c729c7357881\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" Oct 30 19:45:00.628197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.628178 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc67df7-4435-4033-95dd-c729c7357881-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx\" (UID: \"adc67df7-4435-4033-95dd-c729c7357881\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" Oct 30 19:45:00.628281 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.628252 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d140123-d166-4f06-8d0c-ed0bb574f0e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls\" (UID: \"4d140123-d166-4f06-8d0c-ed0bb574f0e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" Oct 30 19:45:00.645334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.645164 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfbt4\" (UniqueName: \"kubernetes.io/projected/adc67df7-4435-4033-95dd-c729c7357881-kube-api-access-vfbt4\") pod \"maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx\" (UID: \"adc67df7-4435-4033-95dd-c729c7357881\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" Oct 30 19:45:00.648778 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.648749 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5csr\" (UniqueName: \"kubernetes.io/projected/4d140123-d166-4f06-8d0c-ed0bb574f0e1-kube-api-access-z5csr\") pod \"maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls\" (UID: \"4d140123-d166-4f06-8d0c-ed0bb574f0e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" Oct 30 19:45:00.699498 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.699465 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" Oct 30 19:45:00.703787 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:00.703758 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" Oct 30 19:45:01.023996 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:45:01.023960 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc67df7_4435_4033_95dd_c729c7357881.slice/crio-988b30be8e33b556a1a5b7fa1259521b761fd66cabf7c9a005c1221bedde88d6 WatchSource:0}: Error finding container 988b30be8e33b556a1a5b7fa1259521b761fd66cabf7c9a005c1221bedde88d6: Status 404 returned error can't find the container with id 988b30be8e33b556a1a5b7fa1259521b761fd66cabf7c9a005c1221bedde88d6 Oct 30 19:45:01.024154 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:01.024127 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx"] Oct 30 19:45:01.041155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:01.041091 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" event={"ID":"adc67df7-4435-4033-95dd-c729c7357881","Type":"ContainerStarted","Data":"988b30be8e33b556a1a5b7fa1259521b761fd66cabf7c9a005c1221bedde88d6"} Oct 30 19:45:01.048369 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:45:01.048335 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d140123_d166_4f06_8d0c_ed0bb574f0e1.slice/crio-4c7843d510dd885914064df7e29e60636fad026ea2bebb778804f8b3d2ce1c5b WatchSource:0}: Error finding container 4c7843d510dd885914064df7e29e60636fad026ea2bebb778804f8b3d2ce1c5b: Status 404 returned error can't find the container with id 4c7843d510dd885914064df7e29e60636fad026ea2bebb778804f8b3d2ce1c5b Oct 30 19:45:01.058074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:01.058026 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a306b8ac-1021-4e38-be1c-aea68485bea3" path="/var/lib/kubelet/pods/a306b8ac-1021-4e38-be1c-aea68485bea3/volumes" Oct 30 19:45:01.058890 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:01.058523 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls"] Oct 30 19:45:02.062975 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:02.062928 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" event={"ID":"4d140123-d166-4f06-8d0c-ed0bb574f0e1","Type":"ContainerStarted","Data":"4c7843d510dd885914064df7e29e60636fad026ea2bebb778804f8b3d2ce1c5b"} Oct 30 19:45:05.099664 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:05.099619 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" event={"ID":"adc67df7-4435-4033-95dd-c729c7357881","Type":"ContainerStarted","Data":"032aa6961e4bacb708deec3fa0175afba3148a3c49947438901ca9ab7f44061f"} Oct 30 19:45:05.101755 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:05.101723 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" event={"ID":"4d140123-d166-4f06-8d0c-ed0bb574f0e1","Type":"ContainerStarted","Data":"a6aa4b9a07690cec7bc2c5da8bf3b7e35153d3dfc3b509246b1f069a563faea2"} Oct 30 19:45:05.144150 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:05.144096 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" podStartSLOduration=1.885322238 podStartE2EDuration="5.144078636s" totalImagesPullingTime="3.258756398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.13863487 +0000 UTC m=+2458.213090229" watchObservedRunningTime="2026-10-30 19:45:05.144078636 +0000 UTC m=+2458.218533991" Oct 30 19:45:05.176700 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:05.176624 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" podStartSLOduration=2.1141971489999998 podStartE2EDuration="5.176605561s" totalImagesPullingTime="3.062408412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.175187617 +0000 UTC m=+2458.249642975" watchObservedRunningTime="2026-10-30 19:45:05.176605561 +0000 UTC m=+2458.251060913" Oct 30 19:45:06.111588 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:06.111548 5047 generic.go:356] "Generic (PLEG): container finished" podID="adc67df7-4435-4033-95dd-c729c7357881" containerID="032aa6961e4bacb708deec3fa0175afba3148a3c49947438901ca9ab7f44061f" exitCode=60 Oct 30 19:45:06.112058 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:06.111755 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" event={"ID":"adc67df7-4435-4033-95dd-c729c7357881","Type":"ContainerDied","Data":"032aa6961e4bacb708deec3fa0175afba3148a3c49947438901ca9ab7f44061f"} Oct 30 19:45:06.112254 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:06.112236 5047 scope.go:122] "RemoveContainer" containerID="032aa6961e4bacb708deec3fa0175afba3148a3c49947438901ca9ab7f44061f" Oct 30 19:45:06.115032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:06.114988 5047 generic.go:356] "Generic (PLEG): container finished" podID="4d140123-d166-4f06-8d0c-ed0bb574f0e1" containerID="a6aa4b9a07690cec7bc2c5da8bf3b7e35153d3dfc3b509246b1f069a563faea2" exitCode=60 Oct 30 19:45:06.115136 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:06.115034 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" event={"ID":"4d140123-d166-4f06-8d0c-ed0bb574f0e1","Type":"ContainerDied","Data":"a6aa4b9a07690cec7bc2c5da8bf3b7e35153d3dfc3b509246b1f069a563faea2"} Oct 30 19:45:06.115374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:06.115356 5047 scope.go:122] "RemoveContainer" containerID="a6aa4b9a07690cec7bc2c5da8bf3b7e35153d3dfc3b509246b1f069a563faea2" Oct 30 19:45:07.125997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:07.125939 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" event={"ID":"adc67df7-4435-4033-95dd-c729c7357881","Type":"ContainerStarted","Data":"a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc"} Oct 30 19:45:07.128326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:07.128296 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" event={"ID":"4d140123-d166-4f06-8d0c-ed0bb574f0e1","Type":"ContainerStarted","Data":"a6c29fdbb0d225181f08a2b7c590ef88605d7b3c552a8755cf67704fa6251a3e"} Oct 30 19:45:08.138245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:08.138206 5047 generic.go:356] "Generic (PLEG): container finished" podID="adc67df7-4435-4033-95dd-c729c7357881" containerID="a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc" exitCode=60 Oct 30 19:45:08.138707 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:08.138284 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" event={"ID":"adc67df7-4435-4033-95dd-c729c7357881","Type":"ContainerDied","Data":"a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc"} Oct 30 19:45:08.138707 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:08.138352 5047 scope.go:122] "RemoveContainer" containerID="032aa6961e4bacb708deec3fa0175afba3148a3c49947438901ca9ab7f44061f" Oct 30 19:45:08.138820 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:08.138774 5047 scope.go:122] "RemoveContainer" containerID="a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc" Oct 30 19:45:08.139281 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:08.139244 5047 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx_odh-ai-gateway-infra(adc67df7-4435-4033-95dd-c729c7357881)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" podUID="adc67df7-4435-4033-95dd-c729c7357881" Oct 30 19:45:08.140715 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:08.140692 5047 generic.go:356] "Generic (PLEG): container finished" podID="4d140123-d166-4f06-8d0c-ed0bb574f0e1" containerID="a6c29fdbb0d225181f08a2b7c590ef88605d7b3c552a8755cf67704fa6251a3e" exitCode=60 Oct 30 19:45:08.140811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:08.140747 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" event={"ID":"4d140123-d166-4f06-8d0c-ed0bb574f0e1","Type":"ContainerDied","Data":"a6c29fdbb0d225181f08a2b7c590ef88605d7b3c552a8755cf67704fa6251a3e"} Oct 30 19:45:08.141103 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:08.141089 5047 scope.go:122] "RemoveContainer" containerID="a6c29fdbb0d225181f08a2b7c590ef88605d7b3c552a8755cf67704fa6251a3e" Oct 30 19:45:08.141445 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:08.141429 5047 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls_odh-ai-gateway-infra(4d140123-d166-4f06-8d0c-ed0bb574f0e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" podUID="4d140123-d166-4f06-8d0c-ed0bb574f0e1" Oct 30 19:45:08.152818 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:08.152797 5047 scope.go:122] "RemoveContainer" containerID="a6aa4b9a07690cec7bc2c5da8bf3b7e35153d3dfc3b509246b1f069a563faea2" Oct 30 19:45:09.010515 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:09.010472 5047 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc67df7_4435_4033_95dd_c729c7357881.slice/crio-a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:45:09.010744 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:09.010522 5047 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc67df7_4435_4033_95dd_c729c7357881.slice/crio-a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:45:09.152111 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:09.152079 5047 scope.go:122] "RemoveContainer" containerID="a6c29fdbb0d225181f08a2b7c590ef88605d7b3c552a8755cf67704fa6251a3e" Oct 30 19:45:09.152573 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:09.152547 5047 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls_odh-ai-gateway-infra(4d140123-d166-4f06-8d0c-ed0bb574f0e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" podUID="4d140123-d166-4f06-8d0c-ed0bb574f0e1" Oct 30 19:45:09.154085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:09.154061 5047 scope.go:122] "RemoveContainer" containerID="a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc" Oct 30 19:45:09.154413 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:09.154392 5047 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx_odh-ai-gateway-infra(adc67df7-4435-4033-95dd-c729c7357881)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" podUID="adc67df7-4435-4033-95dd-c729c7357881" Oct 30 19:45:09.603364 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:09.603324 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:45:10.939160 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:10.939119 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr"] Oct 30 19:45:10.939564 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:10.939374 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" podUID="c4296edd-bfda-4c0d-898a-55bb6c711682" containerName="istio-proxy" containerID="cri-o://eef2ff9876305455713248e00e4c060cae7867cef7ae47f810b6b06d76f64c38" gracePeriod=30 Oct 30 19:45:14.275755 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:14.275714 5047 patch_prober.go:29] interesting pod/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:45:14.276202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:14.275769 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" podUID="c4296edd-bfda-4c0d-898a-55bb6c711682" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:45:16.244210 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.244179 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:45:16.270099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.270069 5047 generic.go:356] "Generic (PLEG): container finished" podID="c4296edd-bfda-4c0d-898a-55bb6c711682" containerID="eef2ff9876305455713248e00e4c060cae7867cef7ae47f810b6b06d76f64c38" exitCode=0 Oct 30 19:45:16.270327 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.270117 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" Oct 30 19:45:16.270327 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.270152 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" event={"ID":"c4296edd-bfda-4c0d-898a-55bb6c711682","Type":"ContainerDied","Data":"eef2ff9876305455713248e00e4c060cae7867cef7ae47f810b6b06d76f64c38"} Oct 30 19:45:16.270327 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.270217 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr" event={"ID":"c4296edd-bfda-4c0d-898a-55bb6c711682","Type":"ContainerDied","Data":"b8c829d8f02057870b3fe9e89c4a738c1d5b2775720c5b936a3a31619b8de8ab"} Oct 30 19:45:16.270327 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.270248 5047 scope.go:122] "RemoveContainer" containerID="eef2ff9876305455713248e00e4c060cae7867cef7ae47f810b6b06d76f64c38" Oct 30 19:45:16.282056 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.282034 5047 scope.go:122] "RemoveContainer" containerID="6463058d753528fe240ad999bc35d4bad4f45e6f06fbbf55e9206ad0c8ad1956" Oct 30 19:45:16.295208 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.295171 5047 scope.go:122] "RemoveContainer" containerID="eef2ff9876305455713248e00e4c060cae7867cef7ae47f810b6b06d76f64c38" Oct 30 19:45:16.295751 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:16.295718 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eef2ff9876305455713248e00e4c060cae7867cef7ae47f810b6b06d76f64c38\": container with ID starting with eef2ff9876305455713248e00e4c060cae7867cef7ae47f810b6b06d76f64c38 not found: ID does not exist" containerID="eef2ff9876305455713248e00e4c060cae7867cef7ae47f810b6b06d76f64c38" Oct 30 19:45:16.295867 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.295762 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef2ff9876305455713248e00e4c060cae7867cef7ae47f810b6b06d76f64c38"} err="failed to get container status \"eef2ff9876305455713248e00e4c060cae7867cef7ae47f810b6b06d76f64c38\": rpc error: code = NotFound desc = could not find container \"eef2ff9876305455713248e00e4c060cae7867cef7ae47f810b6b06d76f64c38\": container with ID starting with eef2ff9876305455713248e00e4c060cae7867cef7ae47f810b6b06d76f64c38 not found: ID does not exist" Oct 30 19:45:16.295867 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.295789 5047 scope.go:122] "RemoveContainer" containerID="6463058d753528fe240ad999bc35d4bad4f45e6f06fbbf55e9206ad0c8ad1956" Oct 30 19:45:16.296474 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:16.296163 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6463058d753528fe240ad999bc35d4bad4f45e6f06fbbf55e9206ad0c8ad1956\": container with ID starting with 6463058d753528fe240ad999bc35d4bad4f45e6f06fbbf55e9206ad0c8ad1956 not found: ID does not exist" containerID="6463058d753528fe240ad999bc35d4bad4f45e6f06fbbf55e9206ad0c8ad1956" Oct 30 19:45:16.296474 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.296195 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6463058d753528fe240ad999bc35d4bad4f45e6f06fbbf55e9206ad0c8ad1956"} err="failed to get container status \"6463058d753528fe240ad999bc35d4bad4f45e6f06fbbf55e9206ad0c8ad1956\": rpc error: code = NotFound desc = could not find container \"6463058d753528fe240ad999bc35d4bad4f45e6f06fbbf55e9206ad0c8ad1956\": container with ID starting with 6463058d753528fe240ad999bc35d4bad4f45e6f06fbbf55e9206ad0c8ad1956 not found: ID does not exist" Oct 30 19:45:16.361304 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.361265 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-token\" (UniqueName: \"kubernetes.io/projected/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-token\") pod \"c4296edd-bfda-4c0d-898a-55bb6c711682\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " Oct 30 19:45:16.361500 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.361321 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4296edd-bfda-4c0d-898a-55bb6c711682-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4296edd-bfda-4c0d-898a-55bb6c711682-istiod-ca-cert\") pod \"c4296edd-bfda-4c0d-898a-55bb6c711682\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " Oct 30 19:45:16.361500 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.361359 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-workload-socket\") pod \"c4296edd-bfda-4c0d-898a-55bb6c711682\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " Oct 30 19:45:16.361500 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.361419 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-data\") pod \"c4296edd-bfda-4c0d-898a-55bb6c711682\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " Oct 30 19:45:16.361651 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.361522 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-envoy\") pod \"c4296edd-bfda-4c0d-898a-55bb6c711682\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " Oct 30 19:45:16.361707 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.361673 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-workload-certs\") pod \"c4296edd-bfda-4c0d-898a-55bb6c711682\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " Oct 30 19:45:16.361794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.361768 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-workload-socket" pod "c4296edd-bfda-4c0d-898a-55bb6c711682" (UID: "c4296edd-bfda-4c0d-898a-55bb6c711682"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:16.361869 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.361774 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-podinfo\") pod \"c4296edd-bfda-4c0d-898a-55bb6c711682\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " Oct 30 19:45:16.361869 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.361812 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4296edd-bfda-4c0d-898a-55bb6c711682-istiod-ca-cert" pod "c4296edd-bfda-4c0d-898a-55bb6c711682" (UID: "c4296edd-bfda-4c0d-898a-55bb6c711682"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:16.362045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.362016 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-credential-socket\") pod \"c4296edd-bfda-4c0d-898a-55bb6c711682\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " Oct 30 19:45:16.362127 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.362088 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4296edd-bfda-4c0d-898a-55bb6c711682-kube-api-access-t94z5\" (UniqueName: \"kubernetes.io/projected/c4296edd-bfda-4c0d-898a-55bb6c711682-kube-api-access-t94z5\") pod \"c4296edd-bfda-4c0d-898a-55bb6c711682\" (UID: \"c4296edd-bfda-4c0d-898a-55bb6c711682\") " Oct 30 19:45:16.362176 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.362118 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-workload-certs" pod "c4296edd-bfda-4c0d-898a-55bb6c711682" (UID: "c4296edd-bfda-4c0d-898a-55bb6c711682"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:16.362329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.362287 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-credential-socket" pod "c4296edd-bfda-4c0d-898a-55bb6c711682" (UID: "c4296edd-bfda-4c0d-898a-55bb6c711682"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:16.362898 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.362871 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:16.362998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.362904 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:16.362998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.362919 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4296edd-bfda-4c0d-898a-55bb6c711682-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:16.362998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.362934 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:16.363250 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.363227 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-data" pod "c4296edd-bfda-4c0d-898a-55bb6c711682" (UID: "c4296edd-bfda-4c0d-898a-55bb6c711682"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:16.363993 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.363968 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-podinfo" pod "c4296edd-bfda-4c0d-898a-55bb6c711682" (UID: "c4296edd-bfda-4c0d-898a-55bb6c711682"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:16.365577 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.365551 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-envoy" pod "c4296edd-bfda-4c0d-898a-55bb6c711682" (UID: "c4296edd-bfda-4c0d-898a-55bb6c711682"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:16.365739 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.365722 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-token" pod "c4296edd-bfda-4c0d-898a-55bb6c711682" (UID: "c4296edd-bfda-4c0d-898a-55bb6c711682"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:16.365933 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.365918 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4296edd-bfda-4c0d-898a-55bb6c711682-kube-api-access-t94z5" pod "c4296edd-bfda-4c0d-898a-55bb6c711682" (UID: "c4296edd-bfda-4c0d-898a-55bb6c711682"). InnerVolumeSpecName "kube-api-access-t94z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:16.464990 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.464896 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:16.464990 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.464933 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:16.464990 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.464945 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t94z5\" (UniqueName: \"kubernetes.io/projected/c4296edd-bfda-4c0d-898a-55bb6c711682-kube-api-access-t94z5\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:16.464990 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.464955 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:16.464990 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.464963 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4296edd-bfda-4c0d-898a-55bb6c711682-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:16.618696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.618661 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr"] Oct 30 19:45:16.626251 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:16.626217 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5d92fa-openshift-default-86bc6d4665-x8wjr"] Oct 30 19:45:17.051790 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:17.051744 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4296edd-bfda-4c0d-898a-55bb6c711682" path="/var/lib/kubelet/pods/c4296edd-bfda-4c0d-898a-55bb6c711682/volumes" Oct 30 19:45:19.061522 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:19.060260 5047 scope.go:122] "RemoveContainer" containerID="a6c29fdbb0d225181f08a2b7c590ef88605d7b3c552a8755cf67704fa6251a3e" Oct 30 19:45:19.215130 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:19.215085 5047 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc67df7_4435_4033_95dd_c729c7357881.slice/crio-a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:45:20.042042 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:20.041877 5047 scope.go:122] "RemoveContainer" containerID="a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc" Oct 30 19:45:20.327138 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:20.327067 5047 generic.go:356] "Generic (PLEG): container finished" podID="4d140123-d166-4f06-8d0c-ed0bb574f0e1" containerID="b9419fd6fa4dad4ea53fe8240f066cdf443d1d7a307363be467c7e578b51226c" exitCode=60 Oct 30 19:45:20.327551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:20.327141 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" event={"ID":"4d140123-d166-4f06-8d0c-ed0bb574f0e1","Type":"ContainerDied","Data":"b9419fd6fa4dad4ea53fe8240f066cdf443d1d7a307363be467c7e578b51226c"} Oct 30 19:45:20.327551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:20.327206 5047 scope.go:122] "RemoveContainer" containerID="a6c29fdbb0d225181f08a2b7c590ef88605d7b3c552a8755cf67704fa6251a3e" Oct 30 19:45:20.327679 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:20.327605 5047 scope.go:122] "RemoveContainer" containerID="b9419fd6fa4dad4ea53fe8240f066cdf443d1d7a307363be467c7e578b51226c" Oct 30 19:45:20.328041 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:20.327999 5047 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls_odh-ai-gateway-infra(4d140123-d166-4f06-8d0c-ed0bb574f0e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" podUID="4d140123-d166-4f06-8d0c-ed0bb574f0e1" Oct 30 19:45:21.339851 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:21.339812 5047 generic.go:356] "Generic (PLEG): container finished" podID="adc67df7-4435-4033-95dd-c729c7357881" containerID="bc8d7a811e8c908adee5d8642840b7e00fb981680cb9de22a1a73201593c1389" exitCode=60 Oct 30 19:45:21.340332 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:21.339883 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" event={"ID":"adc67df7-4435-4033-95dd-c729c7357881","Type":"ContainerDied","Data":"bc8d7a811e8c908adee5d8642840b7e00fb981680cb9de22a1a73201593c1389"} Oct 30 19:45:21.340332 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:21.339941 5047 scope.go:122] "RemoveContainer" containerID="a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc" Oct 30 19:45:21.340443 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:21.340426 5047 scope.go:122] "RemoveContainer" containerID="bc8d7a811e8c908adee5d8642840b7e00fb981680cb9de22a1a73201593c1389" Oct 30 19:45:21.340941 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:21.340911 5047 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx_odh-ai-gateway-infra(adc67df7-4435-4033-95dd-c729c7357881)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" podUID="adc67df7-4435-4033-95dd-c729c7357881" Oct 30 19:45:21.373952 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:21.373917 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls"] Oct 30 19:45:21.575263 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:21.575234 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" Oct 30 19:45:21.690815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:21.690774 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d140123-d166-4f06-8d0c-ed0bb574f0e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d140123-d166-4f06-8d0c-ed0bb574f0e1-maas-api-service-ca\") pod \"4d140123-d166-4f06-8d0c-ed0bb574f0e1\" (UID: \"4d140123-d166-4f06-8d0c-ed0bb574f0e1\") " Oct 30 19:45:21.691068 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:21.691045 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d140123-d166-4f06-8d0c-ed0bb574f0e1-kube-api-access-z5csr\" (UniqueName: \"kubernetes.io/projected/4d140123-d166-4f06-8d0c-ed0bb574f0e1-kube-api-access-z5csr\") pod \"4d140123-d166-4f06-8d0c-ed0bb574f0e1\" (UID: \"4d140123-d166-4f06-8d0c-ed0bb574f0e1\") " Oct 30 19:45:21.691256 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:21.691230 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d140123-d166-4f06-8d0c-ed0bb574f0e1-maas-api-service-ca" pod "4d140123-d166-4f06-8d0c-ed0bb574f0e1" (UID: "4d140123-d166-4f06-8d0c-ed0bb574f0e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:21.691741 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:21.691716 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d140123-d166-4f06-8d0c-ed0bb574f0e1-maas-api-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:21.693271 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:21.693250 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d140123-d166-4f06-8d0c-ed0bb574f0e1-kube-api-access-z5csr" pod "4d140123-d166-4f06-8d0c-ed0bb574f0e1" (UID: "4d140123-d166-4f06-8d0c-ed0bb574f0e1"). InnerVolumeSpecName "kube-api-access-z5csr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:21.793066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:21.792970 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5csr\" (UniqueName: \"kubernetes.io/projected/4d140123-d166-4f06-8d0c-ed0bb574f0e1-kube-api-access-z5csr\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:22.376106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:22.376072 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" event={"ID":"4d140123-d166-4f06-8d0c-ed0bb574f0e1","Type":"ContainerDied","Data":"4c7843d510dd885914064df7e29e60636fad026ea2bebb778804f8b3d2ce1c5b"} Oct 30 19:45:22.376647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:22.376605 5047 scope.go:122] "RemoveContainer" containerID="b9419fd6fa4dad4ea53fe8240f066cdf443d1d7a307363be467c7e578b51226c" Oct 30 19:45:22.380782 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:22.380764 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls" Oct 30 19:45:22.391337 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:22.391307 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx"] Oct 30 19:45:22.472866 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:22.472832 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls"] Oct 30 19:45:22.486102 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:22.482890 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9efc72-29889825-8lpls"] Oct 30 19:45:22.602180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:22.602145 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" Oct 30 19:45:22.619576 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:22.619539 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc67df7-4435-4033-95dd-c729c7357881-kube-api-access-vfbt4\" (UniqueName: \"kubernetes.io/projected/adc67df7-4435-4033-95dd-c729c7357881-kube-api-access-vfbt4\") pod \"adc67df7-4435-4033-95dd-c729c7357881\" (UID: \"adc67df7-4435-4033-95dd-c729c7357881\") " Oct 30 19:45:22.619755 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:22.619743 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adc67df7-4435-4033-95dd-c729c7357881-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc67df7-4435-4033-95dd-c729c7357881-maas-api-service-ca\") pod \"adc67df7-4435-4033-95dd-c729c7357881\" (UID: \"adc67df7-4435-4033-95dd-c729c7357881\") " Oct 30 19:45:22.620200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:22.620168 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc67df7-4435-4033-95dd-c729c7357881-maas-api-service-ca" pod "adc67df7-4435-4033-95dd-c729c7357881" (UID: "adc67df7-4435-4033-95dd-c729c7357881"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:22.620342 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:22.620310 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc67df7-4435-4033-95dd-c729c7357881-maas-api-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:22.621898 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:22.621853 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc67df7-4435-4033-95dd-c729c7357881-kube-api-access-vfbt4" pod "adc67df7-4435-4033-95dd-c729c7357881" (UID: "adc67df7-4435-4033-95dd-c729c7357881"). InnerVolumeSpecName "kube-api-access-vfbt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:22.729999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:22.729945 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfbt4\" (UniqueName: \"kubernetes.io/projected/adc67df7-4435-4033-95dd-c729c7357881-kube-api-access-vfbt4\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:23.081609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:23.080300 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d140123-d166-4f06-8d0c-ed0bb574f0e1" path="/var/lib/kubelet/pods/4d140123-d166-4f06-8d0c-ed0bb574f0e1/volumes" Oct 30 19:45:23.390326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:23.390240 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" Oct 30 19:45:23.390326 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:23.390285 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx" event={"ID":"adc67df7-4435-4033-95dd-c729c7357881","Type":"ContainerDied","Data":"988b30be8e33b556a1a5b7fa1259521b761fd66cabf7c9a005c1221bedde88d6"} Oct 30 19:45:23.390842 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:23.390351 5047 scope.go:122] "RemoveContainer" containerID="bc8d7a811e8c908adee5d8642840b7e00fb981680cb9de22a1a73201593c1389" Oct 30 19:45:23.437283 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:23.437250 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx"] Oct 30 19:45:23.446799 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:23.446772 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09feeb-29889825-qc7wx"] Oct 30 19:45:23.968105 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:23.967979 5047 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc67df7_4435_4033_95dd_c729c7357881.slice/crio-a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:45:24.265254 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:24.265169 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl"] Oct 30 19:45:24.265436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:24.265409 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" podUID="75e6cb8a-1ee5-4a80-a28f-e146bd24f649" containerName="maas-api" containerID="cri-o://92d8b2e1055a22f4fcb2af663590569960a9e99cc9b335aa729cd07cb3b57ff3" gracePeriod=30 Oct 30 19:45:24.404808 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:24.404773 5047 generic.go:356] "Generic (PLEG): container finished" podID="75e6cb8a-1ee5-4a80-a28f-e146bd24f649" containerID="92d8b2e1055a22f4fcb2af663590569960a9e99cc9b335aa729cd07cb3b57ff3" exitCode=0 Oct 30 19:45:24.405265 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:24.404844 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" event={"ID":"75e6cb8a-1ee5-4a80-a28f-e146bd24f649","Type":"ContainerDied","Data":"92d8b2e1055a22f4fcb2af663590569960a9e99cc9b335aa729cd07cb3b57ff3"} Oct 30 19:45:24.935652 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:24.935598 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:45:25.008693 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.008655 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-maas-api-tls\") pod \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " Oct 30 19:45:25.008897 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.008795 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-openshift-service-ca\") pod \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " Oct 30 19:45:25.008979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.008896 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-metrics-certs\") pod \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " Oct 30 19:45:25.008979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.008940 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-kube-api-access-hc6rz\" (UniqueName: \"kubernetes.io/projected/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-kube-api-access-hc6rz\") pod \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\" (UID: \"75e6cb8a-1ee5-4a80-a28f-e146bd24f649\") " Oct 30 19:45:25.009258 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.009225 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-openshift-service-ca" pod "75e6cb8a-1ee5-4a80-a28f-e146bd24f649" (UID: "75e6cb8a-1ee5-4a80-a28f-e146bd24f649"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:25.009683 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.009660 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:25.011060 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.011026 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-maas-api-tls" pod "75e6cb8a-1ee5-4a80-a28f-e146bd24f649" (UID: "75e6cb8a-1ee5-4a80-a28f-e146bd24f649"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:25.011197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.011177 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-kube-api-access-hc6rz" pod "75e6cb8a-1ee5-4a80-a28f-e146bd24f649" (UID: "75e6cb8a-1ee5-4a80-a28f-e146bd24f649"). InnerVolumeSpecName "kube-api-access-hc6rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:25.011421 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.011404 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-metrics-certs" pod "75e6cb8a-1ee5-4a80-a28f-e146bd24f649" (UID: "75e6cb8a-1ee5-4a80-a28f-e146bd24f649"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:25.056098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.056062 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adc67df7-4435-4033-95dd-c729c7357881" path="/var/lib/kubelet/pods/adc67df7-4435-4033-95dd-c729c7357881/volumes" Oct 30 19:45:25.112200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.112114 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:25.112200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.112147 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc6rz\" (UniqueName: \"kubernetes.io/projected/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-kube-api-access-hc6rz\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:25.112200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.112161 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e6cb8a-1ee5-4a80-a28f-e146bd24f649-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:25.441955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.441862 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" Oct 30 19:45:25.441955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.441905 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl" event={"ID":"75e6cb8a-1ee5-4a80-a28f-e146bd24f649","Type":"ContainerDied","Data":"c0a7f8bc9532f386d0480ff6bb7037d2968ef381c4793e20dd05ef900e343a9a"} Oct 30 19:45:25.442529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.441977 5047 scope.go:122] "RemoveContainer" containerID="92d8b2e1055a22f4fcb2af663590569960a9e99cc9b335aa729cd07cb3b57ff3" Oct 30 19:45:25.491186 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.491146 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl"] Oct 30 19:45:25.495696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:25.495668 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-469724-59f488d86b-ghrgl"] Oct 30 19:45:27.060654 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:27.060594 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e6cb8a-1ee5-4a80-a28f-e146bd24f649" path="/var/lib/kubelet/pods/75e6cb8a-1ee5-4a80-a28f-e146bd24f649/volumes" Oct 30 19:45:29.316727 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:29.316693 5047 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc67df7_4435_4033_95dd_c729c7357881.slice/crio-a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:45:32.751278 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.751234 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9"] Oct 30 19:45:32.752516 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.752489 5047 cpu_manager.go:414] "containerMap: removing container" podUID="c4296edd-bfda-4c0d-898a-55bb6c711682" containerName="istio-proxy" Oct 30 19:45:32.752516 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.752515 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4296edd-bfda-4c0d-898a-55bb6c711682" containerName="istio-proxy" Oct 30 19:45:32.752745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.752533 5047 cpu_manager.go:414] "containerMap: removing container" podUID="c4296edd-bfda-4c0d-898a-55bb6c711682" containerName="istio-proxy" Oct 30 19:45:32.752745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.752541 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4296edd-bfda-4c0d-898a-55bb6c711682" containerName="istio-proxy" Oct 30 19:45:32.752745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.752561 5047 cpu_manager.go:414] "containerMap: removing container" podUID="4d140123-d166-4f06-8d0c-ed0bb574f0e1" containerName="cleanup" Oct 30 19:45:32.752745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.752570 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d140123-d166-4f06-8d0c-ed0bb574f0e1" containerName="cleanup" Oct 30 19:45:32.752745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.752617 5047 cpu_manager.go:414] "containerMap: removing container" podUID="adc67df7-4435-4033-95dd-c729c7357881" containerName="cleanup" Oct 30 19:45:32.752745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.752644 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc67df7-4435-4033-95dd-c729c7357881" containerName="cleanup" Oct 30 19:45:32.752745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.752653 5047 cpu_manager.go:414] "containerMap: removing container" podUID="adc67df7-4435-4033-95dd-c729c7357881" containerName="cleanup" Oct 30 19:45:32.752745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.752660 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc67df7-4435-4033-95dd-c729c7357881" containerName="cleanup" Oct 30 19:45:32.752745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.752708 5047 cpu_manager.go:414] "containerMap: removing container" podUID="4d140123-d166-4f06-8d0c-ed0bb574f0e1" containerName="cleanup" Oct 30 19:45:32.752745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.752715 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d140123-d166-4f06-8d0c-ed0bb574f0e1" containerName="cleanup" Oct 30 19:45:32.752745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.752736 5047 cpu_manager.go:414] "containerMap: removing container" podUID="75e6cb8a-1ee5-4a80-a28f-e146bd24f649" containerName="maas-api" Oct 30 19:45:32.752745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.752743 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e6cb8a-1ee5-4a80-a28f-e146bd24f649" containerName="maas-api" Oct 30 19:45:32.753479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.752962 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d140123-d166-4f06-8d0c-ed0bb574f0e1" containerName="cleanup" Oct 30 19:45:32.753479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.752976 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc67df7-4435-4033-95dd-c729c7357881" containerName="cleanup" Oct 30 19:45:32.753479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.752984 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc67df7-4435-4033-95dd-c729c7357881" containerName="cleanup" Oct 30 19:45:32.753479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.753017 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d140123-d166-4f06-8d0c-ed0bb574f0e1" containerName="cleanup" Oct 30 19:45:32.753479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.753034 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e6cb8a-1ee5-4a80-a28f-e146bd24f649" containerName="maas-api" Oct 30 19:45:32.753479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.753086 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4296edd-bfda-4c0d-898a-55bb6c711682" containerName="istio-proxy" Oct 30 19:45:32.753479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.753097 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4296edd-bfda-4c0d-898a-55bb6c711682" containerName="istio-proxy" Oct 30 19:45:32.760185 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.760151 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:32.776874 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.776703 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9"] Oct 30 19:45:32.781575 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.781544 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1f6cef\"/\"openshift-service-ca.crt\"" Oct 30 19:45:32.781575 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.781564 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1f6cef\"/\"kube-root-ca.crt\"" Oct 30 19:45:32.783237 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.783212 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1f6cef\"/\"default-dockercfg-d7m55\"" Oct 30 19:45:32.783355 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.783344 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1f6cef\"/\"test-model-1f6cef-kserve-self-signed-certs\"" Oct 30 19:45:32.902884 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.902843 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-dshm\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:32.903110 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.903040 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-model-cache\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:32.903205 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.903172 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-tmp-dir\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:32.903266 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.903250 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b774b412-c5c8-49cc-b504-4b7418fead88-tls-certs\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:32.903330 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.903315 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-home\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:32.903394 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:32.903383 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86qjr\" (UniqueName: \"kubernetes.io/projected/b774b412-c5c8-49cc-b504-4b7418fead88-kube-api-access-86qjr\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:33.013083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:33.007090 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-home\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:33.013083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:33.007199 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86qjr\" (UniqueName: \"kubernetes.io/projected/b774b412-c5c8-49cc-b504-4b7418fead88-kube-api-access-86qjr\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:33.013083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:33.007316 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-dshm\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:33.013083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:33.007447 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-model-cache\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:33.013083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:33.007540 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-tmp-dir\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:33.013083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:33.007570 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b774b412-c5c8-49cc-b504-4b7418fead88-tls-certs\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:33.049025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:33.043254 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-model-cache\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:33.049025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:33.043553 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-tmp-dir\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:33.049025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:33.043790 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-home\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:33.079713 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:33.067739 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86qjr\" (UniqueName: \"kubernetes.io/projected/b774b412-c5c8-49cc-b504-4b7418fead88-kube-api-access-86qjr\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:33.079713 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:33.079639 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b774b412-c5c8-49cc-b504-4b7418fead88-tls-certs\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:33.102371 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:33.102333 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-dshm\") pod \"test-model-1f6cef-kserve-5dc44d5c4f-zb4m9\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:33.384087 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:33.383774 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:33.687713 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:33.687671 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9"] Oct 30 19:45:33.687931 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:45:33.687909 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb774b412_c5c8_49cc_b504_4b7418fead88.slice/crio-2acd6497fef8a2c1980a9340c04d72fa25137e8121c9bc601bc6d8ed6dfabfb8 WatchSource:0}: Error finding container 2acd6497fef8a2c1980a9340c04d72fa25137e8121c9bc601bc6d8ed6dfabfb8: Status 404 returned error can't find the container with id 2acd6497fef8a2c1980a9340c04d72fa25137e8121c9bc601bc6d8ed6dfabfb8 Oct 30 19:45:34.035962 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.035869 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722"] Oct 30 19:45:34.037162 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.037136 5047 cpu_manager.go:414] "containerMap: removing container" podUID="4d140123-d166-4f06-8d0c-ed0bb574f0e1" containerName="cleanup" Oct 30 19:45:34.037162 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.037162 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d140123-d166-4f06-8d0c-ed0bb574f0e1" containerName="cleanup" Oct 30 19:45:34.037315 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.037254 5047 cpu_manager.go:414] "containerMap: removing container" podUID="adc67df7-4435-4033-95dd-c729c7357881" containerName="cleanup" Oct 30 19:45:34.037315 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.037262 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc67df7-4435-4033-95dd-c729c7357881" containerName="cleanup" Oct 30 19:45:34.037447 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.037434 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc67df7-4435-4033-95dd-c729c7357881" containerName="cleanup" Oct 30 19:45:34.037483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.037457 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d140123-d166-4f06-8d0c-ed0bb574f0e1" containerName="cleanup" Oct 30 19:45:34.044532 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.044506 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 19:45:34.059315 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.059035 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722"] Oct 30 19:45:34.065311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.065260 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-870193-serving-cert\"" Oct 30 19:45:34.139846 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.139809 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-466dm\" (UniqueName: \"kubernetes.io/projected/87cc3e48-0e55-4a0a-ac53-e586f8098a35-kube-api-access-466dm\") pod \"maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722\" (UID: \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 19:45:34.140041 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.139860 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87cc3e48-0e55-4a0a-ac53-e586f8098a35-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722\" (UID: \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 19:45:34.140088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.140039 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87cc3e48-0e55-4a0a-ac53-e586f8098a35-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722\" (UID: \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 19:45:34.140170 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.140153 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87cc3e48-0e55-4a0a-ac53-e586f8098a35-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722\" (UID: \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 19:45:34.251217 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.243381 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-466dm\" (UniqueName: \"kubernetes.io/projected/87cc3e48-0e55-4a0a-ac53-e586f8098a35-kube-api-access-466dm\") pod \"maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722\" (UID: \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 19:45:34.251708 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.251675 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87cc3e48-0e55-4a0a-ac53-e586f8098a35-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722\" (UID: \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 19:45:34.252062 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:34.251976 5047 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-serving-cert: secret "maas-api-e2e-shared-b-w4-870193-serving-cert" not found Oct 30 19:45:34.253555 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:34.253518 5047 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87cc3e48-0e55-4a0a-ac53-e586f8098a35-maas-api-tls podName:87cc3e48-0e55-4a0a-ac53-e586f8098a35 nodeName:}" failed. No retries permitted until 2026-10-30 19:45:34.753493786 +0000 UTC m=+2487.827949143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/87cc3e48-0e55-4a0a-ac53-e586f8098a35-maas-api-tls") pod "maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" (UID: "87cc3e48-0e55-4a0a-ac53-e586f8098a35") : secret "maas-api-e2e-shared-b-w4-870193-serving-cert" not found Oct 30 19:45:34.253732 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.253024 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87cc3e48-0e55-4a0a-ac53-e586f8098a35-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722\" (UID: \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 19:45:34.253891 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.252197 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87cc3e48-0e55-4a0a-ac53-e586f8098a35-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722\" (UID: \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 19:45:34.254225 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.254206 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87cc3e48-0e55-4a0a-ac53-e586f8098a35-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722\" (UID: \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 19:45:34.279825 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.279741 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87cc3e48-0e55-4a0a-ac53-e586f8098a35-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722\" (UID: \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 19:45:34.297077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.294536 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-466dm\" (UniqueName: \"kubernetes.io/projected/87cc3e48-0e55-4a0a-ac53-e586f8098a35-kube-api-access-466dm\") pod \"maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722\" (UID: \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 19:45:34.569985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.569883 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" event={"ID":"b774b412-c5c8-49cc-b504-4b7418fead88","Type":"ContainerStarted","Data":"4877dfecfa704e94a76740dcf9636659e465b06179d79bcb14be8d57002733c5"} Oct 30 19:45:34.569985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.569941 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" event={"ID":"b774b412-c5c8-49cc-b504-4b7418fead88","Type":"ContainerStarted","Data":"2acd6497fef8a2c1980a9340c04d72fa25137e8121c9bc601bc6d8ed6dfabfb8"} Oct 30 19:45:34.616362 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.616297 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" podStartSLOduration=2.616276963 podStartE2EDuration="2.616276963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:34.610484562 +0000 UTC m=+2487.684939932" watchObservedRunningTime="2026-10-30 19:45:34.616276963 +0000 UTC m=+2487.690732318" Oct 30 19:45:34.781028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.773265 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87cc3e48-0e55-4a0a-ac53-e586f8098a35-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722\" (UID: \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 19:45:34.813659 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:34.813621 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87cc3e48-0e55-4a0a-ac53-e586f8098a35-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722\" (UID: \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 19:45:35.004027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:35.000251 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 19:45:35.329902 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:45:35.329869 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87cc3e48_0e55_4a0a_ac53_e586f8098a35.slice/crio-78505714ccd9eec60d9333de953a3c6116aee3884aad1e4c8320d803af884154 WatchSource:0}: Error finding container 78505714ccd9eec60d9333de953a3c6116aee3884aad1e4c8320d803af884154: Status 404 returned error can't find the container with id 78505714ccd9eec60d9333de953a3c6116aee3884aad1e4c8320d803af884154 Oct 30 19:45:35.329902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:35.329885 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722"] Oct 30 19:45:35.387032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:35.384125 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:35.579422 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:35.579387 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" event={"ID":"87cc3e48-0e55-4a0a-ac53-e586f8098a35","Type":"ContainerStarted","Data":"78505714ccd9eec60d9333de953a3c6116aee3884aad1e4c8320d803af884154"} Oct 30 19:45:36.589753 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:36.589706 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" event={"ID":"87cc3e48-0e55-4a0a-ac53-e586f8098a35","Type":"ContainerStarted","Data":"4711068e91739c83461729ebc55a4d25a4a793ced4ca68c259fe31fdd9a4f297"} Oct 30 19:45:36.590264 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:36.589804 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 19:45:36.631937 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:36.631869 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" podStartSLOduration=2.240498307 podStartE2EDuration="2.631852903s" totalImagesPullingTime="391.354596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:36.627435052 +0000 UTC m=+2489.701890409" watchObservedRunningTime="2026-10-30 19:45:36.631852903 +0000 UTC m=+2489.706308263" Oct 30 19:45:39.044348 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:39.044311 5047 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc67df7_4435_4033_95dd_c729c7357881.slice/crio-a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:45:39.336349 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:39.336308 5047 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc67df7_4435_4033_95dd_c729c7357881.slice/crio-a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:45:39.604853 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:39.604763 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:45:42.603787 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:42.603753 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 19:45:43.384835 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:43.384795 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:43.396989 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:43.396960 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:43.659227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:43.659143 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:45:49.480205 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:49.480161 5047 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc67df7_4435_4033_95dd_c729c7357881.slice/crio-a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:45:52.695512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:52.695475 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8"] Oct 30 19:45:52.696041 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:52.695698 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" podUID="074a1938-2380-4eb4-adf4-a15e3162507f" containerName="istio-proxy" containerID="cri-o://1205eab80be08fefc58e3adf333e00b60a3985a3e3592125529b10fe242640b4" gracePeriod=30 Oct 30 19:45:52.707839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:52.707796 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26"] Oct 30 19:45:52.708356 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:52.708318 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" podUID="c724ef66-462d-429a-8b12-c6b9d7f90a1e" containerName="istio-proxy" containerID="cri-o://cbbb0054ba2d99fed97e6b3b86d973d24452d560205251334d1460098a2e0ebc" gracePeriod=30 Oct 30 19:45:53.967463 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:53.967426 5047 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc67df7_4435_4033_95dd_c729c7357881.slice/crio-a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:45:55.001203 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:55.001158 5047 patch_prober.go:29] interesting pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:55.001639 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:55.001238 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" podUID="df7012d4-5d6b-4586-a4ac-629154c1d347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:55.778561 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:55.778525 5047 generic.go:356] "Generic (PLEG): container finished" podID="df7012d4-5d6b-4586-a4ac-629154c1d347" containerID="6bfa287be7a9eddd82860bbb2bb4a305fb81803b776a95b7c127f09a31a7f7f6" exitCode=0 Oct 30 19:45:55.778780 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:55.778597 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" event={"ID":"df7012d4-5d6b-4586-a4ac-629154c1d347","Type":"ContainerDied","Data":"6bfa287be7a9eddd82860bbb2bb4a305fb81803b776a95b7c127f09a31a7f7f6"} Oct 30 19:45:55.779168 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:55.779145 5047 scope.go:122] "RemoveContainer" containerID="6bfa287be7a9eddd82860bbb2bb4a305fb81803b776a95b7c127f09a31a7f7f6" Oct 30 19:45:55.781181 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:55.781153 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:45:55.781246 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:55.781231 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:45:55.781287 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:55.781262 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:45:56.788350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:56.788314 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" event={"ID":"df7012d4-5d6b-4586-a4ac-629154c1d347","Type":"ContainerStarted","Data":"7663aa77ca4c9b8e5823ae42f1ecb32daaf4303d1bde9a086a06e1b92f97baf1"} Oct 30 19:45:57.122078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:57.121986 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:45:57.122413 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:57.122389 5047 patch_prober.go:29] interesting pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 19:45:57.122470 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:57.122438 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" podUID="df7012d4-5d6b-4586-a4ac-629154c1d347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 19:45:58.092443 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.092405 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:45:58.122020 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.121966 5047 patch_prober.go:29] interesting pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 19:45:58.122236 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.122049 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" podUID="df7012d4-5d6b-4586-a4ac-629154c1d347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 19:45:58.198224 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.198188 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:45:58.231785 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.231746 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/074a1938-2380-4eb4-adf4-a15e3162507f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/074a1938-2380-4eb4-adf4-a15e3162507f-istiod-ca-cert\") pod \"074a1938-2380-4eb4-adf4-a15e3162507f\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " Oct 30 19:45:58.231964 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.231804 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-istio-data\") pod \"074a1938-2380-4eb4-adf4-a15e3162507f\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " Oct 30 19:45:58.231964 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.231943 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074a1938-2380-4eb4-adf4-a15e3162507f-kube-api-access-vkjgw\" (UniqueName: \"kubernetes.io/projected/074a1938-2380-4eb4-adf4-a15e3162507f-kube-api-access-vkjgw\") pod \"074a1938-2380-4eb4-adf4-a15e3162507f\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " Oct 30 19:45:58.232181 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.232164 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074a1938-2380-4eb4-adf4-a15e3162507f-istio-token\" (UniqueName: \"kubernetes.io/projected/074a1938-2380-4eb4-adf4-a15e3162507f-istio-token\") pod \"074a1938-2380-4eb4-adf4-a15e3162507f\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " Oct 30 19:45:58.232257 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.232202 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-istio-envoy\") pod \"074a1938-2380-4eb4-adf4-a15e3162507f\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " Oct 30 19:45:58.232347 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.232256 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-istio-data" pod "074a1938-2380-4eb4-adf4-a15e3162507f" (UID: "074a1938-2380-4eb4-adf4-a15e3162507f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:58.232347 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.232308 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074a1938-2380-4eb4-adf4-a15e3162507f-istiod-ca-cert" pod "074a1938-2380-4eb4-adf4-a15e3162507f" (UID: "074a1938-2380-4eb4-adf4-a15e3162507f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:58.232347 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.232333 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-credential-socket\") pod \"074a1938-2380-4eb4-adf4-a15e3162507f\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " Oct 30 19:45:58.232517 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.232457 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-workload-certs\") pod \"074a1938-2380-4eb4-adf4-a15e3162507f\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " Oct 30 19:45:58.232580 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.232532 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/074a1938-2380-4eb4-adf4-a15e3162507f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/074a1938-2380-4eb4-adf4-a15e3162507f-istio-podinfo\") pod \"074a1938-2380-4eb4-adf4-a15e3162507f\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " Oct 30 19:45:58.232580 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.232572 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-workload-socket\") pod \"074a1938-2380-4eb4-adf4-a15e3162507f\" (UID: \"074a1938-2380-4eb4-adf4-a15e3162507f\") " Oct 30 19:45:58.256590 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.232745 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-credential-socket" pod "074a1938-2380-4eb4-adf4-a15e3162507f" (UID: "074a1938-2380-4eb4-adf4-a15e3162507f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:58.256590 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.232966 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-workload-socket" pod "074a1938-2380-4eb4-adf4-a15e3162507f" (UID: "074a1938-2380-4eb4-adf4-a15e3162507f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:58.256590 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.233248 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-workload-certs" pod "074a1938-2380-4eb4-adf4-a15e3162507f" (UID: "074a1938-2380-4eb4-adf4-a15e3162507f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:58.256590 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.233453 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.256590 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.233476 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.256590 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.233493 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.256590 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.233509 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/074a1938-2380-4eb4-adf4-a15e3162507f-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.256590 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.233522 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.267734 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.265297 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074a1938-2380-4eb4-adf4-a15e3162507f-istio-token" pod "074a1938-2380-4eb4-adf4-a15e3162507f" (UID: "074a1938-2380-4eb4-adf4-a15e3162507f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:58.267734 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.265640 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-istio-envoy" pod "074a1938-2380-4eb4-adf4-a15e3162507f" (UID: "074a1938-2380-4eb4-adf4-a15e3162507f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:58.269766 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.268044 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/074a1938-2380-4eb4-adf4-a15e3162507f-istio-podinfo" pod "074a1938-2380-4eb4-adf4-a15e3162507f" (UID: "074a1938-2380-4eb4-adf4-a15e3162507f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:58.269766 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.268088 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074a1938-2380-4eb4-adf4-a15e3162507f-kube-api-access-vkjgw" pod "074a1938-2380-4eb4-adf4-a15e3162507f" (UID: "074a1938-2380-4eb4-adf4-a15e3162507f"). InnerVolumeSpecName "kube-api-access-vkjgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.334184 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-podinfo\") pod \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.334326 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-workload-certs\") pod \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.334376 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-data\") pod \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.334415 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-token\" (UniqueName: \"kubernetes.io/projected/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-token\") pod \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.334517 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c724ef66-462d-429a-8b12-c6b9d7f90a1e-kube-api-access-fwlr4\" (UniqueName: \"kubernetes.io/projected/c724ef66-462d-429a-8b12-c6b9d7f90a1e-kube-api-access-fwlr4\") pod \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.334563 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-envoy\") pod \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.334592 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istiod-ca-cert\") pod \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.334644 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-workload-socket\") pod \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.334708 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-credential-socket\") pod \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\" (UID: \"c724ef66-462d-429a-8b12-c6b9d7f90a1e\") " Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.335538 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/074a1938-2380-4eb4-adf4-a15e3162507f-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.335563 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkjgw\" (UniqueName: \"kubernetes.io/projected/074a1938-2380-4eb4-adf4-a15e3162507f-kube-api-access-vkjgw\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.335578 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/074a1938-2380-4eb4-adf4-a15e3162507f-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.335593 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/074a1938-2380-4eb4-adf4-a15e3162507f-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.379281 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-workload-socket" pod "c724ef66-462d-429a-8b12-c6b9d7f90a1e" (UID: "c724ef66-462d-429a-8b12-c6b9d7f90a1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.379615 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istiod-ca-cert" pod "c724ef66-462d-429a-8b12-c6b9d7f90a1e" (UID: "c724ef66-462d-429a-8b12-c6b9d7f90a1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.383156 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-workload-certs" pod "c724ef66-462d-429a-8b12-c6b9d7f90a1e" (UID: "c724ef66-462d-429a-8b12-c6b9d7f90a1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.392638 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-podinfo" pod "c724ef66-462d-429a-8b12-c6b9d7f90a1e" (UID: "c724ef66-462d-429a-8b12-c6b9d7f90a1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.394135 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-envoy" pod "c724ef66-462d-429a-8b12-c6b9d7f90a1e" (UID: "c724ef66-462d-429a-8b12-c6b9d7f90a1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:58.394334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.394285 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-credential-socket" pod "c724ef66-462d-429a-8b12-c6b9d7f90a1e" (UID: "c724ef66-462d-429a-8b12-c6b9d7f90a1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:58.399028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.398491 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-data" pod "c724ef66-462d-429a-8b12-c6b9d7f90a1e" (UID: "c724ef66-462d-429a-8b12-c6b9d7f90a1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:58.416688 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.404412 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-token" pod "c724ef66-462d-429a-8b12-c6b9d7f90a1e" (UID: "c724ef66-462d-429a-8b12-c6b9d7f90a1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:58.442926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.438325 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c724ef66-462d-429a-8b12-c6b9d7f90a1e-kube-api-access-fwlr4" pod "c724ef66-462d-429a-8b12-c6b9d7f90a1e" (UID: "c724ef66-462d-429a-8b12-c6b9d7f90a1e"). InnerVolumeSpecName "kube-api-access-fwlr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:58.442926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.442737 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.442926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.442770 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.442926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.442786 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.442926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.442798 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.442926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.442811 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwlr4\" (UniqueName: \"kubernetes.io/projected/c724ef66-462d-429a-8b12-c6b9d7f90a1e-kube-api-access-fwlr4\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.442926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.442827 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.442926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.442842 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c724ef66-462d-429a-8b12-c6b9d7f90a1e-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.442926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.442854 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.442926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.442867 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c724ef66-462d-429a-8b12-c6b9d7f90a1e-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:45:58.807121 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.807079 5047 generic.go:356] "Generic (PLEG): container finished" podID="074a1938-2380-4eb4-adf4-a15e3162507f" containerID="1205eab80be08fefc58e3adf333e00b60a3985a3e3592125529b10fe242640b4" exitCode=0 Oct 30 19:45:58.807287 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.807129 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" Oct 30 19:45:58.807287 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.807162 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" event={"ID":"074a1938-2380-4eb4-adf4-a15e3162507f","Type":"ContainerDied","Data":"1205eab80be08fefc58e3adf333e00b60a3985a3e3592125529b10fe242640b4"} Oct 30 19:45:58.807287 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.807217 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8" event={"ID":"074a1938-2380-4eb4-adf4-a15e3162507f","Type":"ContainerDied","Data":"a0bf2528eb718ef2826371c88266e1ffb85cc418793a9260c25c6a16b25ba1a1"} Oct 30 19:45:58.807287 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.807242 5047 scope.go:122] "RemoveContainer" containerID="1205eab80be08fefc58e3adf333e00b60a3985a3e3592125529b10fe242640b4" Oct 30 19:45:58.808909 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.808883 5047 generic.go:356] "Generic (PLEG): container finished" podID="c724ef66-462d-429a-8b12-c6b9d7f90a1e" containerID="cbbb0054ba2d99fed97e6b3b86d973d24452d560205251334d1460098a2e0ebc" exitCode=0 Oct 30 19:45:58.809055 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.808957 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" Oct 30 19:45:58.809055 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.808972 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" event={"ID":"c724ef66-462d-429a-8b12-c6b9d7f90a1e","Type":"ContainerDied","Data":"cbbb0054ba2d99fed97e6b3b86d973d24452d560205251334d1460098a2e0ebc"} Oct 30 19:45:58.809160 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.809070 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26" event={"ID":"c724ef66-462d-429a-8b12-c6b9d7f90a1e","Type":"ContainerDied","Data":"187dbf2725895d6aca224201dc248d1089b12662f9c0930d969219d6d518164a"} Oct 30 19:45:58.822748 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.822608 5047 scope.go:122] "RemoveContainer" containerID="1205eab80be08fefc58e3adf333e00b60a3985a3e3592125529b10fe242640b4" Oct 30 19:45:58.822994 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:58.822971 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1205eab80be08fefc58e3adf333e00b60a3985a3e3592125529b10fe242640b4\": container with ID starting with 1205eab80be08fefc58e3adf333e00b60a3985a3e3592125529b10fe242640b4 not found: ID does not exist" containerID="1205eab80be08fefc58e3adf333e00b60a3985a3e3592125529b10fe242640b4" Oct 30 19:45:58.823064 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.823019 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1205eab80be08fefc58e3adf333e00b60a3985a3e3592125529b10fe242640b4"} err="failed to get container status \"1205eab80be08fefc58e3adf333e00b60a3985a3e3592125529b10fe242640b4\": rpc error: code = NotFound desc = could not find container \"1205eab80be08fefc58e3adf333e00b60a3985a3e3592125529b10fe242640b4\": container with ID starting with 1205eab80be08fefc58e3adf333e00b60a3985a3e3592125529b10fe242640b4 not found: ID does not exist" Oct 30 19:45:58.823064 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.823043 5047 scope.go:122] "RemoveContainer" containerID="cbbb0054ba2d99fed97e6b3b86d973d24452d560205251334d1460098a2e0ebc" Oct 30 19:45:58.833881 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.833856 5047 scope.go:122] "RemoveContainer" containerID="cbbb0054ba2d99fed97e6b3b86d973d24452d560205251334d1460098a2e0ebc" Oct 30 19:45:58.862231 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:58.862185 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbbb0054ba2d99fed97e6b3b86d973d24452d560205251334d1460098a2e0ebc\": container with ID starting with cbbb0054ba2d99fed97e6b3b86d973d24452d560205251334d1460098a2e0ebc not found: ID does not exist" containerID="cbbb0054ba2d99fed97e6b3b86d973d24452d560205251334d1460098a2e0ebc" Oct 30 19:45:58.862501 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.862237 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbbb0054ba2d99fed97e6b3b86d973d24452d560205251334d1460098a2e0ebc"} err="failed to get container status \"cbbb0054ba2d99fed97e6b3b86d973d24452d560205251334d1460098a2e0ebc\": rpc error: code = NotFound desc = could not find container \"cbbb0054ba2d99fed97e6b3b86d973d24452d560205251334d1460098a2e0ebc\": container with ID starting with cbbb0054ba2d99fed97e6b3b86d973d24452d560205251334d1460098a2e0ebc not found: ID does not exist" Oct 30 19:45:58.866957 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.866871 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8"] Oct 30 19:45:58.876075 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.876038 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-s2nd8"] Oct 30 19:45:58.908174 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.908137 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26"] Oct 30 19:45:58.912349 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:58.912318 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-x4r26"] Oct 30 19:45:59.072673 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:59.072585 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="074a1938-2380-4eb4-adf4-a15e3162507f" path="/var/lib/kubelet/pods/074a1938-2380-4eb4-adf4-a15e3162507f/volumes" Oct 30 19:45:59.073264 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:59.073242 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c724ef66-462d-429a-8b12-c6b9d7f90a1e" path="/var/lib/kubelet/pods/c724ef66-462d-429a-8b12-c6b9d7f90a1e/volumes" Oct 30 19:45:59.121690 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:59.121658 5047 patch_prober.go:29] interesting pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 19:45:59.122134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:45:59.121730 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" podUID="df7012d4-5d6b-4586-a4ac-629154c1d347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 19:45:59.604695 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:45:59.604649 5047 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc67df7_4435_4033_95dd_c729c7357881.slice/crio-a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:46:00.121737 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:00.121699 5047 patch_prober.go:29] interesting pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 19:46:00.122180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:00.121769 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" podUID="df7012d4-5d6b-4586-a4ac-629154c1d347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 19:46:01.121984 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:01.121948 5047 patch_prober.go:29] interesting pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 19:46:01.122465 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:01.122035 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" podUID="df7012d4-5d6b-4586-a4ac-629154c1d347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 19:46:01.126993 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:46:01.126964 5047 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc67df7_4435_4033_95dd_c729c7357881.slice/crio-a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:46:01.127474 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:46:01.127451 5047 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc67df7_4435_4033_95dd_c729c7357881.slice/crio-a469c23fff6c16fb951dcef5bd2b41ef89d578f520c6ee3f78614f09230dc0fc.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:46:02.206888 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:02.206849 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:46:02.208176 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:02.208143 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:46:02.211044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:02.210981 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:46:08.369889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.369844 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9"] Oct 30 19:46:08.371188 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.371164 5047 cpu_manager.go:414] "containerMap: removing container" podUID="c724ef66-462d-429a-8b12-c6b9d7f90a1e" containerName="istio-proxy" Oct 30 19:46:08.371266 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.371193 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c724ef66-462d-429a-8b12-c6b9d7f90a1e" containerName="istio-proxy" Oct 30 19:46:08.371305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.371288 5047 cpu_manager.go:414] "containerMap: removing container" podUID="074a1938-2380-4eb4-adf4-a15e3162507f" containerName="istio-proxy" Oct 30 19:46:08.371305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.371298 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="074a1938-2380-4eb4-adf4-a15e3162507f" containerName="istio-proxy" Oct 30 19:46:08.371509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.371495 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="c724ef66-462d-429a-8b12-c6b9d7f90a1e" containerName="istio-proxy" Oct 30 19:46:08.371545 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.371526 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="074a1938-2380-4eb4-adf4-a15e3162507f" containerName="istio-proxy" Oct 30 19:46:08.377444 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.377416 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.394249 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.394213 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9"] Oct 30 19:46:08.396158 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.396128 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a56cb5\"/\"openshift-service-ca.crt\"" Oct 30 19:46:08.397097 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.397068 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a56cb5\"/\"kube-root-ca.crt\"" Oct 30 19:46:08.400802 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.400771 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a56cb5\"/\"rate-test-model-a56cb5-kserve-self-signed-certs\"" Oct 30 19:46:08.400934 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.400868 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a56cb5\"/\"default-dockercfg-zxpd6\"" Oct 30 19:46:08.449900 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.449856 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-dshm\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.450117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.449990 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmbz5\" (UniqueName: \"kubernetes.io/projected/038833d2-c645-45ff-861f-64af9e82bae3-kube-api-access-jmbz5\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.450213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.450193 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-home\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.450257 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.450238 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038833d2-c645-45ff-861f-64af9e82bae3-tls-certs\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.450398 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.450373 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-tmp-dir\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.450460 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.450445 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-model-cache\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.552398 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.552355 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-dshm\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.552619 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.552411 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmbz5\" (UniqueName: \"kubernetes.io/projected/038833d2-c645-45ff-861f-64af9e82bae3-kube-api-access-jmbz5\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.552619 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.552507 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-home\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.552619 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.552548 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038833d2-c645-45ff-861f-64af9e82bae3-tls-certs\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.552619 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.552611 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-tmp-dir\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.552879 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.552666 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-model-cache\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.552940 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.552917 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-home\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.553074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.553051 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-tmp-dir\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.553162 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.553122 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-model-cache\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.554793 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.554771 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-dshm\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.555119 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.555102 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038833d2-c645-45ff-861f-64af9e82bae3-tls-certs\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.569045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.569018 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmbz5\" (UniqueName: \"kubernetes.io/projected/038833d2-c645-45ff-861f-64af9e82bae3-kube-api-access-jmbz5\") pod \"rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:08.699455 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:08.699359 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:09.105519 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:09.093772 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9"] Oct 30 19:46:09.136471 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:46:09.136215 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod038833d2_c645_45ff_861f_64af9e82bae3.slice/crio-de9265ec1ae455faf4cfdae48069e838c7c2a9594f829a5488dc29c8b757ef66 WatchSource:0}: Error finding container de9265ec1ae455faf4cfdae48069e838c7c2a9594f829a5488dc29c8b757ef66: Status 404 returned error can't find the container with id de9265ec1ae455faf4cfdae48069e838c7c2a9594f829a5488dc29c8b757ef66 Oct 30 19:46:09.604188 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:46:09.604145 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:46:09.943569 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:09.943473 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" event={"ID":"038833d2-c645-45ff-861f-64af9e82bae3","Type":"ContainerStarted","Data":"57a983f32920588b01f8e294cdf18c13e4db672161a0b3d846f6264bce0bbae8"} Oct 30 19:46:09.943569 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:09.943527 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" event={"ID":"038833d2-c645-45ff-861f-64af9e82bae3","Type":"ContainerStarted","Data":"de9265ec1ae455faf4cfdae48069e838c7c2a9594f829a5488dc29c8b757ef66"} Oct 30 19:46:09.979144 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:09.979082 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" podStartSLOduration=1.9790673509999999 podStartE2EDuration="1.979067351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:09.977447707 +0000 UTC m=+2523.051903063" watchObservedRunningTime="2026-10-30 19:46:09.979067351 +0000 UTC m=+2523.053522710" Oct 30 19:46:10.700574 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:10.700536 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:18.700781 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:18.700746 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:18.714395 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:18.714358 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:19.052786 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:19.052696 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:46:20.931917 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:20.931881 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w"] Oct 30 19:46:20.938605 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:20.938580 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:20.956687 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:20.956652 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w"] Oct 30 19:46:20.957971 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:20.957936 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-tmp-dir\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:20.958108 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:20.958050 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-dshm\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:20.958196 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:20.958178 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-model-cache\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:20.958242 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:20.958226 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d7ca442-33fe-4785-ae1f-f48829771fc0-tls-certs\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:20.958398 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:20.958376 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-home\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:20.958474 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:20.958429 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bztwz\" (UniqueName: \"kubernetes.io/projected/9d7ca442-33fe-4785-ae1f-f48829771fc0-kube-api-access-bztwz\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:20.958779 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:20.958755 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-095f55\"/\"kube-root-ca.crt\"" Oct 30 19:46:20.958844 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:20.958812 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-095f55\"/\"openshift-service-ca.crt\"" Oct 30 19:46:20.967399 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:20.967377 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-095f55\"/\"default-dockercfg-jpj75\"" Oct 30 19:46:20.967399 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:20.967394 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-095f55\"/\"auth-test-model-095f55-kserve-self-signed-certs\"" Oct 30 19:46:21.060459 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:21.060418 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-tmp-dir\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:21.060675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:21.060471 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-dshm\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:21.060675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:21.060585 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-model-cache\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:21.060675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:21.060631 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d7ca442-33fe-4785-ae1f-f48829771fc0-tls-certs\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:21.060854 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:21.060785 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-home\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:21.060854 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:21.060832 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bztwz\" (UniqueName: \"kubernetes.io/projected/9d7ca442-33fe-4785-ae1f-f48829771fc0-kube-api-access-bztwz\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:21.060969 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:21.060879 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-tmp-dir\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:21.061247 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:21.061210 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-home\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:21.061367 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:21.061264 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-model-cache\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:21.063285 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:21.063260 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-dshm\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:21.063530 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:21.063510 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d7ca442-33fe-4785-ae1f-f48829771fc0-tls-certs\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:21.079083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:21.079055 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bztwz\" (UniqueName: \"kubernetes.io/projected/9d7ca442-33fe-4785-ae1f-f48829771fc0-kube-api-access-bztwz\") pod \"auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:21.268548 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:21.266487 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:21.628928 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:21.628891 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w"] Oct 30 19:46:21.633776 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:46:21.633104 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d7ca442_33fe_4785_ae1f_f48829771fc0.slice/crio-0e0788fbfd89282f706c9a3da7849aba49bdea331829e9dcca5d46405c8059a0 WatchSource:0}: Error finding container 0e0788fbfd89282f706c9a3da7849aba49bdea331829e9dcca5d46405c8059a0: Status 404 returned error can't find the container with id 0e0788fbfd89282f706c9a3da7849aba49bdea331829e9dcca5d46405c8059a0 Oct 30 19:46:22.063341 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:22.063232 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" event={"ID":"9d7ca442-33fe-4785-ae1f-f48829771fc0","Type":"ContainerStarted","Data":"288f830b6022fe8083beb1162e7e1aa1709d62570207686c1abd0d364fe252bf"} Oct 30 19:46:22.063341 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:22.063296 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" event={"ID":"9d7ca442-33fe-4785-ae1f-f48829771fc0","Type":"ContainerStarted","Data":"0e0788fbfd89282f706c9a3da7849aba49bdea331829e9dcca5d46405c8059a0"} Oct 30 19:46:22.100256 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:22.100202 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" podStartSLOduration=2.100185036 podStartE2EDuration="2.100185036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:22.097156603 +0000 UTC m=+2535.171611962" watchObservedRunningTime="2026-10-30 19:46:22.100185036 +0000 UTC m=+2535.174640396" Oct 30 19:46:22.273037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:22.272972 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:31.272660 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:31.272625 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:31.286039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:31.285987 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:31.298103 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:31.298077 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:46:36.025953 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:36.025918 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54"] Oct 30 19:46:39.603047 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:46:39.602995 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:46:41.404200 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:41.404163 5047 generic.go:356] "Generic (PLEG): container finished" podID="a05dd723-2d08-4261-aa55-e642430737e6" containerID="854d2769d396b31c54c5370335d46309f17672d4c1b0dbd59b793ab0e5c754a9" exitCode=0 Oct 30 19:46:41.404702 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:41.404236 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" event={"ID":"a05dd723-2d08-4261-aa55-e642430737e6","Type":"ContainerDied","Data":"854d2769d396b31c54c5370335d46309f17672d4c1b0dbd59b793ab0e5c754a9"} Oct 30 19:46:41.404702 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:41.404678 5047 scope.go:122] "RemoveContainer" containerID="854d2769d396b31c54c5370335d46309f17672d4c1b0dbd59b793ab0e5c754a9" Oct 30 19:46:41.406643 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:41.406617 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:46:41.406717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:41.406685 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:46:41.406755 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:41.406719 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:46:42.413764 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:42.413726 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" event={"ID":"a05dd723-2d08-4261-aa55-e642430737e6","Type":"ContainerStarted","Data":"c80360e7660b98bb62fa9d4a2fd75c41716c4e035aedb99982847b68562af31b"} Oct 30 19:46:43.180361 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:43.180325 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:46:43.180596 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:43.180542 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 19:46:43.180680 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:43.180606 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" podUID="a05dd723-2d08-4261-aa55-e642430737e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 19:46:44.180623 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:44.180570 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 19:46:44.181090 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:44.180655 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" podUID="a05dd723-2d08-4261-aa55-e642430737e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 19:46:45.180130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:45.180093 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 19:46:45.180316 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:45.180165 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" podUID="a05dd723-2d08-4261-aa55-e642430737e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 19:46:46.180613 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:46.180564 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 19:46:46.181098 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:46.180631 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" podUID="a05dd723-2d08-4261-aa55-e642430737e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 19:46:47.180557 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:47.180520 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 19:46:47.180758 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:47.180613 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" podUID="a05dd723-2d08-4261-aa55-e642430737e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 19:46:48.186031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:48.185925 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:46:48.186486 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:48.186458 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:46:48.187243 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:48.187221 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 19:46:50.258368 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:50.258320 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv"] Oct 30 19:46:51.060563 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.060517 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" podUID="79c14c0f-27b4-4124-8d40-2e4a0ea85a6e" containerName="main" containerID="cri-o://af2323d6a7516e3afd54a21062af79043c10207f850030182b791a528f181ea7" gracePeriod=45 Oct 30 19:46:51.376475 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.376445 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:46:51.468647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.468596 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-tmp-dir\") pod \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " Oct 30 19:46:51.468848 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.468676 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-tls-certs\" (UniqueName: \"kubernetes.io/secret/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-tls-certs\") pod \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " Oct 30 19:46:51.468930 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.468849 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-kube-api-access-xkfht\" (UniqueName: \"kubernetes.io/projected/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-kube-api-access-xkfht\") pod \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " Oct 30 19:46:51.468996 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.468937 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-dshm\") pod \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " Oct 30 19:46:51.468996 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.468948 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-tmp-dir" pod "79c14c0f-27b4-4124-8d40-2e4a0ea85a6e" (UID: "79c14c0f-27b4-4124-8d40-2e4a0ea85a6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:51.468996 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.468986 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-model-cache\") pod \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " Oct 30 19:46:51.469803 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.469208 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-home\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-home\") pod \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\" (UID: \"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e\") " Oct 30 19:46:51.469803 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.469659 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-model-cache" pod "79c14c0f-27b4-4124-8d40-2e4a0ea85a6e" (UID: "79c14c0f-27b4-4124-8d40-2e4a0ea85a6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:51.469982 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.469958 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-home" pod "79c14c0f-27b4-4124-8d40-2e4a0ea85a6e" (UID: "79c14c0f-27b4-4124-8d40-2e4a0ea85a6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:51.472186 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.471908 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:46:51.472186 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.471938 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:46:51.472186 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.471951 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:46:51.483352 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.483312 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-dshm" pod "79c14c0f-27b4-4124-8d40-2e4a0ea85a6e" (UID: "79c14c0f-27b4-4124-8d40-2e4a0ea85a6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:51.483504 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.483377 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-kube-api-access-xkfht" pod "79c14c0f-27b4-4124-8d40-2e4a0ea85a6e" (UID: "79c14c0f-27b4-4124-8d40-2e4a0ea85a6e"). InnerVolumeSpecName "kube-api-access-xkfht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:51.483571 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.483533 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-tls-certs" pod "79c14c0f-27b4-4124-8d40-2e4a0ea85a6e" (UID: "79c14c0f-27b4-4124-8d40-2e4a0ea85a6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:51.541600 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.541564 5047 generic.go:356] "Generic (PLEG): container finished" podID="79c14c0f-27b4-4124-8d40-2e4a0ea85a6e" containerID="af2323d6a7516e3afd54a21062af79043c10207f850030182b791a528f181ea7" exitCode=0 Oct 30 19:46:51.541806 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.541615 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" Oct 30 19:46:51.541806 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.541654 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" event={"ID":"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e","Type":"ContainerDied","Data":"af2323d6a7516e3afd54a21062af79043c10207f850030182b791a528f181ea7"} Oct 30 19:46:51.541806 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.541708 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54" event={"ID":"79c14c0f-27b4-4124-8d40-2e4a0ea85a6e","Type":"ContainerDied","Data":"5db7a73025e5d20541f23e0bb1cab16ba4280dd545ea835fcd65412535bb1363"} Oct 30 19:46:51.541806 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.541735 5047 scope.go:122] "RemoveContainer" containerID="af2323d6a7516e3afd54a21062af79043c10207f850030182b791a528f181ea7" Oct 30 19:46:51.554080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.554053 5047 scope.go:122] "RemoveContainer" containerID="af2323d6a7516e3afd54a21062af79043c10207f850030182b791a528f181ea7" Oct 30 19:46:51.554432 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:46:51.554407 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af2323d6a7516e3afd54a21062af79043c10207f850030182b791a528f181ea7\": container with ID starting with af2323d6a7516e3afd54a21062af79043c10207f850030182b791a528f181ea7 not found: ID does not exist" containerID="af2323d6a7516e3afd54a21062af79043c10207f850030182b791a528f181ea7" Oct 30 19:46:51.554545 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.554438 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af2323d6a7516e3afd54a21062af79043c10207f850030182b791a528f181ea7"} err="failed to get container status \"af2323d6a7516e3afd54a21062af79043c10207f850030182b791a528f181ea7\": rpc error: code = NotFound desc = could not find container \"af2323d6a7516e3afd54a21062af79043c10207f850030182b791a528f181ea7\": container with ID starting with af2323d6a7516e3afd54a21062af79043c10207f850030182b791a528f181ea7 not found: ID does not exist" Oct 30 19:46:51.573493 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.573457 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:46:51.573493 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.573491 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkfht\" (UniqueName: \"kubernetes.io/projected/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-kube-api-access-xkfht\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:46:51.573703 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.573504 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:46:51.590674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.590504 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54"] Oct 30 19:46:51.594583 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:51.594555 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-08bcfe/test-model-08bcfe-kserve-74456787ff-25p54"] Oct 30 19:46:53.052890 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:46:53.052858 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c14c0f-27b4-4124-8d40-2e4a0ea85a6e" path="/var/lib/kubelet/pods/79c14c0f-27b4-4124-8d40-2e4a0ea85a6e/volumes" Oct 30 19:47:01.023658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:01.023614 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr"] Oct 30 19:47:01.067908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:01.067871 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g"] Oct 30 19:47:05.285058 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.285014 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" podUID="c183d261-57b7-4c90-817e-b4cedd105113" containerName="main" containerID="cri-o://74c4cbb5697a3d81058c7809061bc93d69ee51e645bded81a215ee2482de0746" gracePeriod=45 Oct 30 19:47:05.629531 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.629505 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:47:05.690043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.689989 5047 generic.go:356] "Generic (PLEG): container finished" podID="c183d261-57b7-4c90-817e-b4cedd105113" containerID="74c4cbb5697a3d81058c7809061bc93d69ee51e645bded81a215ee2482de0746" exitCode=0 Oct 30 19:47:05.690235 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.690043 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" event={"ID":"c183d261-57b7-4c90-817e-b4cedd105113","Type":"ContainerDied","Data":"74c4cbb5697a3d81058c7809061bc93d69ee51e645bded81a215ee2482de0746"} Oct 30 19:47:05.690235 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.690070 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" Oct 30 19:47:05.690235 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.690097 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv" event={"ID":"c183d261-57b7-4c90-817e-b4cedd105113","Type":"ContainerDied","Data":"7dc57f846419c88eda7860a95ffd323408109f5d8a71e9284a4510274015be9f"} Oct 30 19:47:05.690235 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.690118 5047 scope.go:122] "RemoveContainer" containerID="74c4cbb5697a3d81058c7809061bc93d69ee51e645bded81a215ee2482de0746" Oct 30 19:47:05.707019 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.706982 5047 scope.go:122] "RemoveContainer" containerID="74c4cbb5697a3d81058c7809061bc93d69ee51e645bded81a215ee2482de0746" Oct 30 19:47:05.707497 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:47:05.707466 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74c4cbb5697a3d81058c7809061bc93d69ee51e645bded81a215ee2482de0746\": container with ID starting with 74c4cbb5697a3d81058c7809061bc93d69ee51e645bded81a215ee2482de0746 not found: ID does not exist" containerID="74c4cbb5697a3d81058c7809061bc93d69ee51e645bded81a215ee2482de0746" Oct 30 19:47:05.707608 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.707506 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74c4cbb5697a3d81058c7809061bc93d69ee51e645bded81a215ee2482de0746"} err="failed to get container status \"74c4cbb5697a3d81058c7809061bc93d69ee51e645bded81a215ee2482de0746\": rpc error: code = NotFound desc = could not find container \"74c4cbb5697a3d81058c7809061bc93d69ee51e645bded81a215ee2482de0746\": container with ID starting with 74c4cbb5697a3d81058c7809061bc93d69ee51e645bded81a215ee2482de0746 not found: ID does not exist" Oct 30 19:47:05.710141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.710101 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-dshm\") pod \"c183d261-57b7-4c90-817e-b4cedd105113\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " Oct 30 19:47:05.710305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.710223 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c183d261-57b7-4c90-817e-b4cedd105113-tls-certs\" (UniqueName: \"kubernetes.io/secret/c183d261-57b7-4c90-817e-b4cedd105113-tls-certs\") pod \"c183d261-57b7-4c90-817e-b4cedd105113\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " Oct 30 19:47:05.710397 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.710323 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-model-cache\") pod \"c183d261-57b7-4c90-817e-b4cedd105113\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " Oct 30 19:47:05.710397 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.710381 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c183d261-57b7-4c90-817e-b4cedd105113-kube-api-access-qj25q\" (UniqueName: \"kubernetes.io/projected/c183d261-57b7-4c90-817e-b4cedd105113-kube-api-access-qj25q\") pod \"c183d261-57b7-4c90-817e-b4cedd105113\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " Oct 30 19:47:05.710586 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.710443 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-tmp-dir\") pod \"c183d261-57b7-4c90-817e-b4cedd105113\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " Oct 30 19:47:05.710586 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.710507 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-home\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-home\") pod \"c183d261-57b7-4c90-817e-b4cedd105113\" (UID: \"c183d261-57b7-4c90-817e-b4cedd105113\") " Oct 30 19:47:05.711294 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.711262 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-tmp-dir" pod "c183d261-57b7-4c90-817e-b4cedd105113" (UID: "c183d261-57b7-4c90-817e-b4cedd105113"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:05.711422 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.711396 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-home" pod "c183d261-57b7-4c90-817e-b4cedd105113" (UID: "c183d261-57b7-4c90-817e-b4cedd105113"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:05.711658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.711632 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-model-cache" pod "c183d261-57b7-4c90-817e-b4cedd105113" (UID: "c183d261-57b7-4c90-817e-b4cedd105113"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:05.713523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.713470 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c183d261-57b7-4c90-817e-b4cedd105113-kube-api-access-qj25q" pod "c183d261-57b7-4c90-817e-b4cedd105113" (UID: "c183d261-57b7-4c90-817e-b4cedd105113"). InnerVolumeSpecName "kube-api-access-qj25q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:05.714475 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.714447 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c183d261-57b7-4c90-817e-b4cedd105113-tls-certs" pod "c183d261-57b7-4c90-817e-b4cedd105113" (UID: "c183d261-57b7-4c90-817e-b4cedd105113"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:05.714585 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.714497 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-dshm" pod "c183d261-57b7-4c90-817e-b4cedd105113" (UID: "c183d261-57b7-4c90-817e-b4cedd105113"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:05.822142 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.815109 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:05.828076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.815222 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c183d261-57b7-4c90-817e-b4cedd105113-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:05.828317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.828106 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:05.828317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.828125 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj25q\" (UniqueName: \"kubernetes.io/projected/c183d261-57b7-4c90-817e-b4cedd105113-kube-api-access-qj25q\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:05.828317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.828139 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:05.828317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:05.828154 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c183d261-57b7-4c90-817e-b4cedd105113-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:06.024929 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:06.024880 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" podUID="77f27bee-11a8-48c1-bdd9-3f76fafd653e" containerName="payload-processing" containerID="cri-o://902e5053fa6f6537851ca9e0e000dd7e2e7268f7c02b0d7908838414cc8288e0" gracePeriod=25 Oct 30 19:47:06.049868 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:06.036951 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv"] Oct 30 19:47:06.049868 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:06.043327 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-09feeb/test-model-09feeb-kserve-595dcf7d45-lt9lv"] Oct 30 19:47:06.068229 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:06.068201 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" podUID="5428f014-f74f-4362-9010-f510a62a5b87" containerName="payload-pre-processing" containerID="cri-o://003ed777e3cbb386c4a1a030051e0ffe4e06212c611737ee6cd128349b85bb4b" gracePeriod=25 Oct 30 19:47:06.358604 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:06.358569 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s"] Oct 30 19:47:07.053833 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:07.053785 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c183d261-57b7-4c90-817e-b4cedd105113" path="/var/lib/kubelet/pods/c183d261-57b7-4c90-817e-b4cedd105113/volumes" Oct 30 19:47:09.603223 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:47:09.603176 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:47:13.934086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:13.933977 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" podUID="5428f014-f74f-4362-9010-f510a62a5b87" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:47:13.934657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:13.933989 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" podUID="77f27bee-11a8-48c1-bdd9-3f76fafd653e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:47:19.752480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:19.752436 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j"] Oct 30 19:47:19.752914 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:19.752713 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" podUID="9aff363b-dce0-48d0-91cc-7783972b1f85" containerName="istio-proxy" containerID="cri-o://5ba6556e27f3d93cb3109c00b6c39811f725ec1ac8c94a2568b21e7bb0830ed6" gracePeriod=30 Oct 30 19:47:21.380814 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.380766 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" podUID="67d13fe4-e24c-4831-80e9-d0994c89ee65" containerName="main" containerID="cri-o://0e9a3461d74a244c21e19f21e39a363cd1cbf0e1f1fd4cdb4e7b8ec693852da6" gracePeriod=45 Oct 30 19:47:21.696868 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.696827 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:47:21.821661 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.821623 5047 generic.go:356] "Generic (PLEG): container finished" podID="67d13fe4-e24c-4831-80e9-d0994c89ee65" containerID="0e9a3461d74a244c21e19f21e39a363cd1cbf0e1f1fd4cdb4e7b8ec693852da6" exitCode=0 Oct 30 19:47:21.821874 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.821667 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" Oct 30 19:47:21.821874 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.821712 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" event={"ID":"67d13fe4-e24c-4831-80e9-d0994c89ee65","Type":"ContainerDied","Data":"0e9a3461d74a244c21e19f21e39a363cd1cbf0e1f1fd4cdb4e7b8ec693852da6"} Oct 30 19:47:21.821874 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.821764 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" event={"ID":"67d13fe4-e24c-4831-80e9-d0994c89ee65","Type":"ContainerDied","Data":"1514c5e2841d521ad811b6afaf8fa9f427ed428654191a09451201db1901e617"} Oct 30 19:47:21.821874 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.821788 5047 scope.go:122] "RemoveContainer" containerID="0e9a3461d74a244c21e19f21e39a363cd1cbf0e1f1fd4cdb4e7b8ec693852da6" Oct 30 19:47:21.828991 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.828964 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-home\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-home\") pod \"67d13fe4-e24c-4831-80e9-d0994c89ee65\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " Oct 30 19:47:21.829155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.829023 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-dshm\") pod \"67d13fe4-e24c-4831-80e9-d0994c89ee65\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " Oct 30 19:47:21.829155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.829123 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-tmp-dir\") pod \"67d13fe4-e24c-4831-80e9-d0994c89ee65\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " Oct 30 19:47:21.829284 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.829263 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-home" pod "67d13fe4-e24c-4831-80e9-d0994c89ee65" (UID: "67d13fe4-e24c-4831-80e9-d0994c89ee65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:21.829395 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.829375 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-model-cache\") pod \"67d13fe4-e24c-4831-80e9-d0994c89ee65\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " Oct 30 19:47:21.829452 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.829420 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d13fe4-e24c-4831-80e9-d0994c89ee65-kube-api-access-dtdt4\" (UniqueName: \"kubernetes.io/projected/67d13fe4-e24c-4831-80e9-d0994c89ee65-kube-api-access-dtdt4\") pod \"67d13fe4-e24c-4831-80e9-d0994c89ee65\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " Oct 30 19:47:21.829515 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.829486 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67d13fe4-e24c-4831-80e9-d0994c89ee65-tls-certs\" (UniqueName: \"kubernetes.io/secret/67d13fe4-e24c-4831-80e9-d0994c89ee65-tls-certs\") pod \"67d13fe4-e24c-4831-80e9-d0994c89ee65\" (UID: \"67d13fe4-e24c-4831-80e9-d0994c89ee65\") " Oct 30 19:47:21.829515 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.829505 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-tmp-dir" pod "67d13fe4-e24c-4831-80e9-d0994c89ee65" (UID: "67d13fe4-e24c-4831-80e9-d0994c89ee65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:21.829746 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.829716 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-model-cache" pod "67d13fe4-e24c-4831-80e9-d0994c89ee65" (UID: "67d13fe4-e24c-4831-80e9-d0994c89ee65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:21.830390 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.830361 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:21.830502 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.830394 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:21.830502 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.830434 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:21.831765 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.831740 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-dshm" pod "67d13fe4-e24c-4831-80e9-d0994c89ee65" (UID: "67d13fe4-e24c-4831-80e9-d0994c89ee65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:21.831900 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.831881 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d13fe4-e24c-4831-80e9-d0994c89ee65-kube-api-access-dtdt4" pod "67d13fe4-e24c-4831-80e9-d0994c89ee65" (UID: "67d13fe4-e24c-4831-80e9-d0994c89ee65"). InnerVolumeSpecName "kube-api-access-dtdt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:21.831947 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.831926 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d13fe4-e24c-4831-80e9-d0994c89ee65-tls-certs" pod "67d13fe4-e24c-4831-80e9-d0994c89ee65" (UID: "67d13fe4-e24c-4831-80e9-d0994c89ee65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:21.834185 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.834161 5047 scope.go:122] "RemoveContainer" containerID="0e9a3461d74a244c21e19f21e39a363cd1cbf0e1f1fd4cdb4e7b8ec693852da6" Oct 30 19:47:21.834533 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:47:21.834510 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e9a3461d74a244c21e19f21e39a363cd1cbf0e1f1fd4cdb4e7b8ec693852da6\": container with ID starting with 0e9a3461d74a244c21e19f21e39a363cd1cbf0e1f1fd4cdb4e7b8ec693852da6 not found: ID does not exist" containerID="0e9a3461d74a244c21e19f21e39a363cd1cbf0e1f1fd4cdb4e7b8ec693852da6" Oct 30 19:47:21.834600 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.834541 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e9a3461d74a244c21e19f21e39a363cd1cbf0e1f1fd4cdb4e7b8ec693852da6"} err="failed to get container status \"0e9a3461d74a244c21e19f21e39a363cd1cbf0e1f1fd4cdb4e7b8ec693852da6\": rpc error: code = NotFound desc = could not find container \"0e9a3461d74a244c21e19f21e39a363cd1cbf0e1f1fd4cdb4e7b8ec693852da6\": container with ID starting with 0e9a3461d74a244c21e19f21e39a363cd1cbf0e1f1fd4cdb4e7b8ec693852da6 not found: ID does not exist" Oct 30 19:47:21.909669 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.909567 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z"] Oct 30 19:47:21.909918 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.909859 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" podUID="484526c0-1f9b-4b29-b26a-192ba5837d58" containerName="maas-api" containerID="cri-o://1e65157962e12e762e772412044b89cc4bc04fc0c0a40755c68570e43772e2f2" gracePeriod=30 Oct 30 19:47:21.932961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.932922 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d13fe4-e24c-4831-80e9-d0994c89ee65-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:21.933171 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.932967 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtdt4\" (UniqueName: \"kubernetes.io/projected/67d13fe4-e24c-4831-80e9-d0994c89ee65-kube-api-access-dtdt4\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:21.933171 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:21.932984 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d13fe4-e24c-4831-80e9-d0994c89ee65-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:22.291170 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.291078 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s"] Oct 30 19:47:22.294076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.293548 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s"] Oct 30 19:47:22.477029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.471690 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:47:22.608115 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.608073 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/484526c0-1f9b-4b29-b26a-192ba5837d58-kube-api-access-8r749\" (UniqueName: \"kubernetes.io/projected/484526c0-1f9b-4b29-b26a-192ba5837d58-kube-api-access-8r749\") pod \"484526c0-1f9b-4b29-b26a-192ba5837d58\" (UID: \"484526c0-1f9b-4b29-b26a-192ba5837d58\") " Oct 30 19:47:22.608338 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.608157 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/484526c0-1f9b-4b29-b26a-192ba5837d58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/484526c0-1f9b-4b29-b26a-192ba5837d58-maas-api-tls\") pod \"484526c0-1f9b-4b29-b26a-192ba5837d58\" (UID: \"484526c0-1f9b-4b29-b26a-192ba5837d58\") " Oct 30 19:47:22.608338 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.608289 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/484526c0-1f9b-4b29-b26a-192ba5837d58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/484526c0-1f9b-4b29-b26a-192ba5837d58-openshift-service-ca\") pod \"484526c0-1f9b-4b29-b26a-192ba5837d58\" (UID: \"484526c0-1f9b-4b29-b26a-192ba5837d58\") " Oct 30 19:47:22.608491 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.608382 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/484526c0-1f9b-4b29-b26a-192ba5837d58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/484526c0-1f9b-4b29-b26a-192ba5837d58-metrics-certs\") pod \"484526c0-1f9b-4b29-b26a-192ba5837d58\" (UID: \"484526c0-1f9b-4b29-b26a-192ba5837d58\") " Oct 30 19:47:22.608894 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.608863 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/484526c0-1f9b-4b29-b26a-192ba5837d58-openshift-service-ca" pod "484526c0-1f9b-4b29-b26a-192ba5837d58" (UID: "484526c0-1f9b-4b29-b26a-192ba5837d58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:22.609192 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.609163 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/484526c0-1f9b-4b29-b26a-192ba5837d58-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:22.610501 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.610471 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/484526c0-1f9b-4b29-b26a-192ba5837d58-maas-api-tls" pod "484526c0-1f9b-4b29-b26a-192ba5837d58" (UID: "484526c0-1f9b-4b29-b26a-192ba5837d58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:22.610658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.610633 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484526c0-1f9b-4b29-b26a-192ba5837d58-kube-api-access-8r749" pod "484526c0-1f9b-4b29-b26a-192ba5837d58" (UID: "484526c0-1f9b-4b29-b26a-192ba5837d58"). InnerVolumeSpecName "kube-api-access-8r749". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:22.610885 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.610868 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/484526c0-1f9b-4b29-b26a-192ba5837d58-metrics-certs" pod "484526c0-1f9b-4b29-b26a-192ba5837d58" (UID: "484526c0-1f9b-4b29-b26a-192ba5837d58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:22.624636 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.624592 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-a7b7bc/auth-test-model-a7b7bc-kserve-54d9f55bbf-7xr6s" podUID="67d13fe4-e24c-4831-80e9-d0994c89ee65" containerName="main" probeResult="failure" output="Get \"https://10.132.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:47:22.710331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.710291 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/484526c0-1f9b-4b29-b26a-192ba5837d58-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:22.710331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.710326 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/484526c0-1f9b-4b29-b26a-192ba5837d58-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:22.710331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.710336 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r749\" (UniqueName: \"kubernetes.io/projected/484526c0-1f9b-4b29-b26a-192ba5837d58-kube-api-access-8r749\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:22.831708 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.831673 5047 generic.go:356] "Generic (PLEG): container finished" podID="484526c0-1f9b-4b29-b26a-192ba5837d58" containerID="1e65157962e12e762e772412044b89cc4bc04fc0c0a40755c68570e43772e2f2" exitCode=0 Oct 30 19:47:22.831895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.831719 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" Oct 30 19:47:22.831895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.831754 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" event={"ID":"484526c0-1f9b-4b29-b26a-192ba5837d58","Type":"ContainerDied","Data":"1e65157962e12e762e772412044b89cc4bc04fc0c0a40755c68570e43772e2f2"} Oct 30 19:47:22.831895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.831811 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z" event={"ID":"484526c0-1f9b-4b29-b26a-192ba5837d58","Type":"ContainerDied","Data":"a3d6f970b0566df096104ef35cca7885e798f4652df9f654d0354544560607da"} Oct 30 19:47:22.831895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.831841 5047 scope.go:122] "RemoveContainer" containerID="1e65157962e12e762e772412044b89cc4bc04fc0c0a40755c68570e43772e2f2" Oct 30 19:47:22.849369 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.849338 5047 scope.go:122] "RemoveContainer" containerID="1e65157962e12e762e772412044b89cc4bc04fc0c0a40755c68570e43772e2f2" Oct 30 19:47:22.849731 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:47:22.849708 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e65157962e12e762e772412044b89cc4bc04fc0c0a40755c68570e43772e2f2\": container with ID starting with 1e65157962e12e762e772412044b89cc4bc04fc0c0a40755c68570e43772e2f2 not found: ID does not exist" containerID="1e65157962e12e762e772412044b89cc4bc04fc0c0a40755c68570e43772e2f2" Oct 30 19:47:22.849803 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.849738 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e65157962e12e762e772412044b89cc4bc04fc0c0a40755c68570e43772e2f2"} err="failed to get container status \"1e65157962e12e762e772412044b89cc4bc04fc0c0a40755c68570e43772e2f2\": rpc error: code = NotFound desc = could not find container \"1e65157962e12e762e772412044b89cc4bc04fc0c0a40755c68570e43772e2f2\": container with ID starting with 1e65157962e12e762e772412044b89cc4bc04fc0c0a40755c68570e43772e2f2 not found: ID does not exist" Oct 30 19:47:22.932786 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.932681 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z"] Oct 30 19:47:22.937466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:22.937432 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08bcfe-5f8b4f6c9f-vlj8z"] Oct 30 19:47:23.052368 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:23.052332 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="484526c0-1f9b-4b29-b26a-192ba5837d58" path="/var/lib/kubelet/pods/484526c0-1f9b-4b29-b26a-192ba5837d58/volumes" Oct 30 19:47:23.052914 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:23.052898 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d13fe4-e24c-4831-80e9-d0994c89ee65" path="/var/lib/kubelet/pods/67d13fe4-e24c-4831-80e9-d0994c89ee65/volumes" Oct 30 19:47:23.808159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:23.808123 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd"] Oct 30 19:47:23.963665 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:23.963145 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" podUID="77f27bee-11a8-48c1-bdd9-3f76fafd653e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:47:23.974324 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:23.974277 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" podUID="5428f014-f74f-4362-9010-f510a62a5b87" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:47:25.104901 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.104873 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:47:25.168166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.168123 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-credential-socket\") pod \"9aff363b-dce0-48d0-91cc-7783972b1f85\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " Oct 30 19:47:25.168358 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.168207 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aff363b-dce0-48d0-91cc-7783972b1f85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aff363b-dce0-48d0-91cc-7783972b1f85-istiod-ca-cert\") pod \"9aff363b-dce0-48d0-91cc-7783972b1f85\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " Oct 30 19:47:25.168358 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.168234 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-workload-certs\") pod \"9aff363b-dce0-48d0-91cc-7783972b1f85\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " Oct 30 19:47:25.168515 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.168400 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-envoy\") pod \"9aff363b-dce0-48d0-91cc-7783972b1f85\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " Oct 30 19:47:25.168515 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.168446 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-credential-socket" pod "9aff363b-dce0-48d0-91cc-7783972b1f85" (UID: "9aff363b-dce0-48d0-91cc-7783972b1f85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:25.168653 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.168548 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-workload-certs" pod "9aff363b-dce0-48d0-91cc-7783972b1f85" (UID: "9aff363b-dce0-48d0-91cc-7783972b1f85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:25.168653 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.168632 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aff363b-dce0-48d0-91cc-7783972b1f85-istiod-ca-cert" pod "9aff363b-dce0-48d0-91cc-7783972b1f85" (UID: "9aff363b-dce0-48d0-91cc-7783972b1f85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:25.168653 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.168637 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-podinfo\") pod \"9aff363b-dce0-48d0-91cc-7783972b1f85\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " Oct 30 19:47:25.168813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.168684 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-token\" (UniqueName: \"kubernetes.io/projected/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-token\") pod \"9aff363b-dce0-48d0-91cc-7783972b1f85\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " Oct 30 19:47:25.168813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.168733 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-data\") pod \"9aff363b-dce0-48d0-91cc-7783972b1f85\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " Oct 30 19:47:25.168935 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.168842 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-workload-socket\") pod \"9aff363b-dce0-48d0-91cc-7783972b1f85\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " Oct 30 19:47:25.168935 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.168922 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aff363b-dce0-48d0-91cc-7783972b1f85-kube-api-access-tvshj\" (UniqueName: \"kubernetes.io/projected/9aff363b-dce0-48d0-91cc-7783972b1f85-kube-api-access-tvshj\") pod \"9aff363b-dce0-48d0-91cc-7783972b1f85\" (UID: \"9aff363b-dce0-48d0-91cc-7783972b1f85\") " Oct 30 19:47:25.169074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.168952 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-data" pod "9aff363b-dce0-48d0-91cc-7783972b1f85" (UID: "9aff363b-dce0-48d0-91cc-7783972b1f85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:25.169284 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.169160 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-workload-socket" pod "9aff363b-dce0-48d0-91cc-7783972b1f85" (UID: "9aff363b-dce0-48d0-91cc-7783972b1f85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:25.169731 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.169710 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:25.169731 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.169731 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:25.169894 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.169740 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:25.169894 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.169749 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aff363b-dce0-48d0-91cc-7783972b1f85-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:25.169894 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.169759 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:25.171210 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.171180 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-envoy" pod "9aff363b-dce0-48d0-91cc-7783972b1f85" (UID: "9aff363b-dce0-48d0-91cc-7783972b1f85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:25.171828 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.171796 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aff363b-dce0-48d0-91cc-7783972b1f85-kube-api-access-tvshj" pod "9aff363b-dce0-48d0-91cc-7783972b1f85" (UID: "9aff363b-dce0-48d0-91cc-7783972b1f85"). InnerVolumeSpecName "kube-api-access-tvshj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:25.172101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.172071 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-token" pod "9aff363b-dce0-48d0-91cc-7783972b1f85" (UID: "9aff363b-dce0-48d0-91cc-7783972b1f85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:25.176163 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.176137 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-podinfo" pod "9aff363b-dce0-48d0-91cc-7783972b1f85" (UID: "9aff363b-dce0-48d0-91cc-7783972b1f85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:47:25.274389 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.272511 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:25.274389 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.272573 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:25.274389 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.272589 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvshj\" (UniqueName: \"kubernetes.io/projected/9aff363b-dce0-48d0-91cc-7783972b1f85-kube-api-access-tvshj\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:25.274389 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.272607 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aff363b-dce0-48d0-91cc-7783972b1f85-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:25.861284 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.861240 5047 generic.go:356] "Generic (PLEG): container finished" podID="9aff363b-dce0-48d0-91cc-7783972b1f85" containerID="5ba6556e27f3d93cb3109c00b6c39811f725ec1ac8c94a2568b21e7bb0830ed6" exitCode=0 Oct 30 19:47:25.861467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.861281 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" event={"ID":"9aff363b-dce0-48d0-91cc-7783972b1f85","Type":"ContainerDied","Data":"5ba6556e27f3d93cb3109c00b6c39811f725ec1ac8c94a2568b21e7bb0830ed6"} Oct 30 19:47:25.861467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.861318 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" Oct 30 19:47:25.861467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.861371 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j" event={"ID":"9aff363b-dce0-48d0-91cc-7783972b1f85","Type":"ContainerDied","Data":"b909da1c97c987498a178541efb4983cb0ad1f0611e818521265cf29479da374"} Oct 30 19:47:25.861467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.861394 5047 scope.go:122] "RemoveContainer" containerID="5ba6556e27f3d93cb3109c00b6c39811f725ec1ac8c94a2568b21e7bb0830ed6" Oct 30 19:47:25.874157 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.873950 5047 scope.go:122] "RemoveContainer" containerID="919f6e2b4a4060e86af745524d5294dd59dabb80f1caf4d3d772f1155fb2a80f" Oct 30 19:47:25.884936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.884917 5047 scope.go:122] "RemoveContainer" containerID="5ba6556e27f3d93cb3109c00b6c39811f725ec1ac8c94a2568b21e7bb0830ed6" Oct 30 19:47:25.885275 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:47:25.885249 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ba6556e27f3d93cb3109c00b6c39811f725ec1ac8c94a2568b21e7bb0830ed6\": container with ID starting with 5ba6556e27f3d93cb3109c00b6c39811f725ec1ac8c94a2568b21e7bb0830ed6 not found: ID does not exist" containerID="5ba6556e27f3d93cb3109c00b6c39811f725ec1ac8c94a2568b21e7bb0830ed6" Oct 30 19:47:25.885391 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.885284 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ba6556e27f3d93cb3109c00b6c39811f725ec1ac8c94a2568b21e7bb0830ed6"} err="failed to get container status \"5ba6556e27f3d93cb3109c00b6c39811f725ec1ac8c94a2568b21e7bb0830ed6\": rpc error: code = NotFound desc = could not find container \"5ba6556e27f3d93cb3109c00b6c39811f725ec1ac8c94a2568b21e7bb0830ed6\": container with ID starting with 5ba6556e27f3d93cb3109c00b6c39811f725ec1ac8c94a2568b21e7bb0830ed6 not found: ID does not exist" Oct 30 19:47:25.885391 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.885302 5047 scope.go:122] "RemoveContainer" containerID="919f6e2b4a4060e86af745524d5294dd59dabb80f1caf4d3d772f1155fb2a80f" Oct 30 19:47:25.885561 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:47:25.885537 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919f6e2b4a4060e86af745524d5294dd59dabb80f1caf4d3d772f1155fb2a80f\": container with ID starting with 919f6e2b4a4060e86af745524d5294dd59dabb80f1caf4d3d772f1155fb2a80f not found: ID does not exist" containerID="919f6e2b4a4060e86af745524d5294dd59dabb80f1caf4d3d772f1155fb2a80f" Oct 30 19:47:25.885608 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.885569 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919f6e2b4a4060e86af745524d5294dd59dabb80f1caf4d3d772f1155fb2a80f"} err="failed to get container status \"919f6e2b4a4060e86af745524d5294dd59dabb80f1caf4d3d772f1155fb2a80f\": rpc error: code = NotFound desc = could not find container \"919f6e2b4a4060e86af745524d5294dd59dabb80f1caf4d3d772f1155fb2a80f\": container with ID starting with 919f6e2b4a4060e86af745524d5294dd59dabb80f1caf4d3d772f1155fb2a80f not found: ID does not exist" Oct 30 19:47:25.923719 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.923681 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j"] Oct 30 19:47:25.933602 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:25.933571 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac614-openshift-default-77488748d7-h8v2j"] Oct 30 19:47:26.165390 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.165352 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz"] Oct 30 19:47:26.166358 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166333 5047 cpu_manager.go:414] "containerMap: removing container" podUID="9aff363b-dce0-48d0-91cc-7783972b1f85" containerName="istio-proxy" Oct 30 19:47:26.166358 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166355 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aff363b-dce0-48d0-91cc-7783972b1f85" containerName="istio-proxy" Oct 30 19:47:26.166358 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166367 5047 cpu_manager.go:414] "containerMap: removing container" podUID="79c14c0f-27b4-4124-8d40-2e4a0ea85a6e" containerName="main" Oct 30 19:47:26.166605 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166372 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c14c0f-27b4-4124-8d40-2e4a0ea85a6e" containerName="main" Oct 30 19:47:26.166605 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166388 5047 cpu_manager.go:414] "containerMap: removing container" podUID="c183d261-57b7-4c90-817e-b4cedd105113" containerName="main" Oct 30 19:47:26.166605 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166394 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c183d261-57b7-4c90-817e-b4cedd105113" containerName="main" Oct 30 19:47:26.166605 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166420 5047 cpu_manager.go:414] "containerMap: removing container" podUID="9aff363b-dce0-48d0-91cc-7783972b1f85" containerName="istio-proxy" Oct 30 19:47:26.166605 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166427 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aff363b-dce0-48d0-91cc-7783972b1f85" containerName="istio-proxy" Oct 30 19:47:26.166605 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166441 5047 cpu_manager.go:414] "containerMap: removing container" podUID="484526c0-1f9b-4b29-b26a-192ba5837d58" containerName="maas-api" Oct 30 19:47:26.166605 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166449 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="484526c0-1f9b-4b29-b26a-192ba5837d58" containerName="maas-api" Oct 30 19:47:26.166605 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166486 5047 cpu_manager.go:414] "containerMap: removing container" podUID="67d13fe4-e24c-4831-80e9-d0994c89ee65" containerName="main" Oct 30 19:47:26.166605 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166492 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d13fe4-e24c-4831-80e9-d0994c89ee65" containerName="main" Oct 30 19:47:26.166889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166659 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="c183d261-57b7-4c90-817e-b4cedd105113" containerName="main" Oct 30 19:47:26.166889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166680 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aff363b-dce0-48d0-91cc-7783972b1f85" containerName="istio-proxy" Oct 30 19:47:26.166889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166689 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="484526c0-1f9b-4b29-b26a-192ba5837d58" containerName="maas-api" Oct 30 19:47:26.166889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166705 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d13fe4-e24c-4831-80e9-d0994c89ee65" containerName="main" Oct 30 19:47:26.166889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166720 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aff363b-dce0-48d0-91cc-7783972b1f85" containerName="istio-proxy" Oct 30 19:47:26.166889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.166732 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c14c0f-27b4-4124-8d40-2e4a0ea85a6e" containerName="main" Oct 30 19:47:26.170921 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.170892 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.187968 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.187925 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-f83282\"/\"kube-root-ca.crt\"" Oct 30 19:47:26.190913 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.190886 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-f83282\"/\"default-dockercfg-5qb9r\"" Oct 30 19:47:26.191197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.190945 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-f83282\"/\"openshift-service-ca.crt\"" Oct 30 19:47:26.191197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.191104 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-f83282\"/\"rate-test-model-f83282-kserve-self-signed-certs\"" Oct 30 19:47:26.216521 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.216437 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz"] Oct 30 19:47:26.299153 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.299112 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-tmp-dir\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.299337 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.299217 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-dshm\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.299337 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.299272 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5dab94b-516a-4db7-9b19-e6f95456abfa-tls-certs\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.299337 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.299311 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-home\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.299459 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.299388 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8f55\" (UniqueName: \"kubernetes.io/projected/f5dab94b-516a-4db7-9b19-e6f95456abfa-kube-api-access-q8f55\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.299459 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.299425 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-model-cache\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.470234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.470191 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" Oct 30 19:47:26.471929 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.471873 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8f55\" (UniqueName: \"kubernetes.io/projected/f5dab94b-516a-4db7-9b19-e6f95456abfa-kube-api-access-q8f55\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.472206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.471941 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-model-cache\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.472287 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.472264 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-tmp-dir\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.472362 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.472341 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-dshm\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.472433 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.472392 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5dab94b-516a-4db7-9b19-e6f95456abfa-tls-certs\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.472433 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.472430 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-home\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.473422 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.472976 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-home\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.473422 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.473349 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-model-cache\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.475119 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.475073 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-tmp-dir\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.518027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.500824 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5dab94b-516a-4db7-9b19-e6f95456abfa-tls-certs\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.570720 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.570685 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8f55\" (UniqueName: \"kubernetes.io/projected/f5dab94b-516a-4db7-9b19-e6f95456abfa-kube-api-access-q8f55\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.573898 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.573860 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f27bee-11a8-48c1-bdd9-3f76fafd653e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f27bee-11a8-48c1-bdd9-3f76fafd653e-plugins-config-volume\") pod \"77f27bee-11a8-48c1-bdd9-3f76fafd653e\" (UID: \"77f27bee-11a8-48c1-bdd9-3f76fafd653e\") " Oct 30 19:47:26.574612 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.574588 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f27bee-11a8-48c1-bdd9-3f76fafd653e-kube-api-access-2mrzk\" (UniqueName: \"kubernetes.io/projected/77f27bee-11a8-48c1-bdd9-3f76fafd653e-kube-api-access-2mrzk\") pod \"77f27bee-11a8-48c1-bdd9-3f76fafd653e\" (UID: \"77f27bee-11a8-48c1-bdd9-3f76fafd653e\") " Oct 30 19:47:26.575218 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.574487 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f27bee-11a8-48c1-bdd9-3f76fafd653e-plugins-config-volume" pod "77f27bee-11a8-48c1-bdd9-3f76fafd653e" (UID: "77f27bee-11a8-48c1-bdd9-3f76fafd653e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:26.577393 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.577098 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f27bee-11a8-48c1-bdd9-3f76fafd653e-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:26.581121 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.581079 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f27bee-11a8-48c1-bdd9-3f76fafd653e-kube-api-access-2mrzk" pod "77f27bee-11a8-48c1-bdd9-3f76fafd653e" (UID: "77f27bee-11a8-48c1-bdd9-3f76fafd653e"). InnerVolumeSpecName "kube-api-access-2mrzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:26.591766 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.591328 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" Oct 30 19:47:26.612677 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.612620 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-dshm\") pod \"rate-test-model-f83282-kserve-756d87c59c-j2lqz\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.678316 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.678189 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5428f014-f74f-4362-9010-f510a62a5b87-kube-api-access-7ljrg\" (UniqueName: \"kubernetes.io/projected/5428f014-f74f-4362-9010-f510a62a5b87-kube-api-access-7ljrg\") pod \"5428f014-f74f-4362-9010-f510a62a5b87\" (UID: \"5428f014-f74f-4362-9010-f510a62a5b87\") " Oct 30 19:47:26.678316 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.678283 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5428f014-f74f-4362-9010-f510a62a5b87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5428f014-f74f-4362-9010-f510a62a5b87-plugins-config-volume\") pod \"5428f014-f74f-4362-9010-f510a62a5b87\" (UID: \"5428f014-f74f-4362-9010-f510a62a5b87\") " Oct 30 19:47:26.678860 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.678827 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5428f014-f74f-4362-9010-f510a62a5b87-plugins-config-volume" pod "5428f014-f74f-4362-9010-f510a62a5b87" (UID: "5428f014-f74f-4362-9010-f510a62a5b87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:26.679069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.679027 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5428f014-f74f-4362-9010-f510a62a5b87-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:26.679169 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.679067 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mrzk\" (UniqueName: \"kubernetes.io/projected/77f27bee-11a8-48c1-bdd9-3f76fafd653e-kube-api-access-2mrzk\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:26.680657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.680622 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5428f014-f74f-4362-9010-f510a62a5b87-kube-api-access-7ljrg" pod "5428f014-f74f-4362-9010-f510a62a5b87" (UID: "5428f014-f74f-4362-9010-f510a62a5b87"). InnerVolumeSpecName "kube-api-access-7ljrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:26.780545 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.780501 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ljrg\" (UniqueName: \"kubernetes.io/projected/5428f014-f74f-4362-9010-f510a62a5b87-kube-api-access-7ljrg\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:26.792916 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.792878 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:26.877587 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.877551 5047 generic.go:356] "Generic (PLEG): container finished" podID="5428f014-f74f-4362-9010-f510a62a5b87" containerID="003ed777e3cbb386c4a1a030051e0ffe4e06212c611737ee6cd128349b85bb4b" exitCode=0 Oct 30 19:47:26.877792 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.877626 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" Oct 30 19:47:26.877792 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.877661 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" event={"ID":"5428f014-f74f-4362-9010-f510a62a5b87","Type":"ContainerDied","Data":"003ed777e3cbb386c4a1a030051e0ffe4e06212c611737ee6cd128349b85bb4b"} Oct 30 19:47:26.877792 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.877715 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g" event={"ID":"5428f014-f74f-4362-9010-f510a62a5b87","Type":"ContainerDied","Data":"7457e1d0e11c3cca165f4e2a429f0cd5b1947d0c77cb1a60d05918aad5f3dd8b"} Oct 30 19:47:26.877792 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.877744 5047 scope.go:122] "RemoveContainer" containerID="003ed777e3cbb386c4a1a030051e0ffe4e06212c611737ee6cd128349b85bb4b" Oct 30 19:47:26.886890 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.884132 5047 generic.go:356] "Generic (PLEG): container finished" podID="77f27bee-11a8-48c1-bdd9-3f76fafd653e" containerID="902e5053fa6f6537851ca9e0e000dd7e2e7268f7c02b0d7908838414cc8288e0" exitCode=0 Oct 30 19:47:26.886890 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.884266 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" event={"ID":"77f27bee-11a8-48c1-bdd9-3f76fafd653e","Type":"ContainerDied","Data":"902e5053fa6f6537851ca9e0e000dd7e2e7268f7c02b0d7908838414cc8288e0"} Oct 30 19:47:26.886890 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.884314 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" event={"ID":"77f27bee-11a8-48c1-bdd9-3f76fafd653e","Type":"ContainerDied","Data":"42124add7a2c2e257790b677dcfa9d0570d1b2cf21f2c8eb7dc66740eb175c17"} Oct 30 19:47:26.886890 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.884395 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr" Oct 30 19:47:26.898174 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.898147 5047 scope.go:122] "RemoveContainer" containerID="003ed777e3cbb386c4a1a030051e0ffe4e06212c611737ee6cd128349b85bb4b" Oct 30 19:47:26.899100 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:47:26.899068 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"003ed777e3cbb386c4a1a030051e0ffe4e06212c611737ee6cd128349b85bb4b\": container with ID starting with 003ed777e3cbb386c4a1a030051e0ffe4e06212c611737ee6cd128349b85bb4b not found: ID does not exist" containerID="003ed777e3cbb386c4a1a030051e0ffe4e06212c611737ee6cd128349b85bb4b" Oct 30 19:47:26.899203 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.899112 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"003ed777e3cbb386c4a1a030051e0ffe4e06212c611737ee6cd128349b85bb4b"} err="failed to get container status \"003ed777e3cbb386c4a1a030051e0ffe4e06212c611737ee6cd128349b85bb4b\": rpc error: code = NotFound desc = could not find container \"003ed777e3cbb386c4a1a030051e0ffe4e06212c611737ee6cd128349b85bb4b\": container with ID starting with 003ed777e3cbb386c4a1a030051e0ffe4e06212c611737ee6cd128349b85bb4b not found: ID does not exist" Oct 30 19:47:26.899203 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.899137 5047 scope.go:122] "RemoveContainer" containerID="902e5053fa6f6537851ca9e0e000dd7e2e7268f7c02b0d7908838414cc8288e0" Oct 30 19:47:26.980210 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.980176 5047 scope.go:122] "RemoveContainer" containerID="902e5053fa6f6537851ca9e0e000dd7e2e7268f7c02b0d7908838414cc8288e0" Oct 30 19:47:26.980895 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:47:26.980865 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902e5053fa6f6537851ca9e0e000dd7e2e7268f7c02b0d7908838414cc8288e0\": container with ID starting with 902e5053fa6f6537851ca9e0e000dd7e2e7268f7c02b0d7908838414cc8288e0 not found: ID does not exist" containerID="902e5053fa6f6537851ca9e0e000dd7e2e7268f7c02b0d7908838414cc8288e0" Oct 30 19:47:26.981086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:26.981060 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902e5053fa6f6537851ca9e0e000dd7e2e7268f7c02b0d7908838414cc8288e0"} err="failed to get container status \"902e5053fa6f6537851ca9e0e000dd7e2e7268f7c02b0d7908838414cc8288e0\": rpc error: code = NotFound desc = could not find container \"902e5053fa6f6537851ca9e0e000dd7e2e7268f7c02b0d7908838414cc8288e0\": container with ID starting with 902e5053fa6f6537851ca9e0e000dd7e2e7268f7c02b0d7908838414cc8288e0 not found: ID does not exist" Oct 30 19:47:27.002524 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:27.002443 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g"] Oct 30 19:47:27.008333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:27.008299 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08bcfe-dbb7f556c-q6m8g"] Oct 30 19:47:27.037636 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:27.037601 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr"] Oct 30 19:47:27.065329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:27.058444 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5428f014-f74f-4362-9010-f510a62a5b87" path="/var/lib/kubelet/pods/5428f014-f74f-4362-9010-f510a62a5b87/volumes" Oct 30 19:47:27.065329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:27.059459 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aff363b-dce0-48d0-91cc-7783972b1f85" path="/var/lib/kubelet/pods/9aff363b-dce0-48d0-91cc-7783972b1f85/volumes" Oct 30 19:47:27.065329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:27.059969 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08bcfe-5454cf5dd-spmkr"] Oct 30 19:47:27.108385 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:27.108355 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz"] Oct 30 19:47:27.109394 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:47:27.109368 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5dab94b_516a_4db7_9b19_e6f95456abfa.slice/crio-7c95b1c6f436c7339b2f5c97f04e1cf1f4147196e261b4003b9e78a4d748fdad WatchSource:0}: Error finding container 7c95b1c6f436c7339b2f5c97f04e1cf1f4147196e261b4003b9e78a4d748fdad: Status 404 returned error can't find the container with id 7c95b1c6f436c7339b2f5c97f04e1cf1f4147196e261b4003b9e78a4d748fdad Oct 30 19:47:27.894360 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:27.894320 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" event={"ID":"f5dab94b-516a-4db7-9b19-e6f95456abfa","Type":"ContainerStarted","Data":"997f45974169c22883bf57a9bf4c7adcece66e2dca25050c8bdf560ed14658a7"} Oct 30 19:47:27.894797 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:27.894367 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" event={"ID":"f5dab94b-516a-4db7-9b19-e6f95456abfa","Type":"ContainerStarted","Data":"7c95b1c6f436c7339b2f5c97f04e1cf1f4147196e261b4003b9e78a4d748fdad"} Oct 30 19:47:27.934281 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:27.934213 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" podStartSLOduration=1.9341963500000001 podStartE2EDuration="1.93419635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:27.929365137 +0000 UTC m=+2601.003820503" watchObservedRunningTime="2026-10-30 19:47:27.93419635 +0000 UTC m=+2601.008651708" Oct 30 19:47:28.418138 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.418104 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn"] Oct 30 19:47:28.418924 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.418902 5047 cpu_manager.go:414] "containerMap: removing container" podUID="5428f014-f74f-4362-9010-f510a62a5b87" containerName="payload-pre-processing" Oct 30 19:47:28.418924 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.418922 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5428f014-f74f-4362-9010-f510a62a5b87" containerName="payload-pre-processing" Oct 30 19:47:28.419136 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.418973 5047 cpu_manager.go:414] "containerMap: removing container" podUID="77f27bee-11a8-48c1-bdd9-3f76fafd653e" containerName="payload-processing" Oct 30 19:47:28.419136 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.418979 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f27bee-11a8-48c1-bdd9-3f76fafd653e" containerName="payload-processing" Oct 30 19:47:28.419136 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.419114 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="5428f014-f74f-4362-9010-f510a62a5b87" containerName="payload-pre-processing" Oct 30 19:47:28.419136 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.419132 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f27bee-11a8-48c1-bdd9-3f76fafd653e" containerName="payload-processing" Oct 30 19:47:28.422867 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.422844 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.441833 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.441792 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn"] Oct 30 19:47:28.446322 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.446287 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-870193\"/\"kube-root-ca.crt\"" Oct 30 19:47:28.446322 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.446315 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-870193\"/\"auth-test-model-870193-kserve-self-signed-certs\"" Oct 30 19:47:28.449172 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.448575 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-870193\"/\"openshift-service-ca.crt\"" Oct 30 19:47:28.449172 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.448575 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-870193\"/\"default-dockercfg-pfhbn\"" Oct 30 19:47:28.503448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.503407 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7735d239-4062-4651-84be-4ea42357260e-tls-certs\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.503650 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.503617 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl4ct\" (UniqueName: \"kubernetes.io/projected/7735d239-4062-4651-84be-4ea42357260e-kube-api-access-bl4ct\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.503745 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.503729 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-tmp-dir\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.503814 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.503801 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-home\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.503879 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.503837 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-dshm\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.504074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.504048 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-model-cache\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.606101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.606056 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-tmp-dir\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.606434 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.606404 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-home\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.627030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.606575 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-dshm\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.627497 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.627460 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-model-cache\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.627686 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.627669 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7735d239-4062-4651-84be-4ea42357260e-tls-certs\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.628068 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.628032 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl4ct\" (UniqueName: \"kubernetes.io/projected/7735d239-4062-4651-84be-4ea42357260e-kube-api-access-bl4ct\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.630069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.630044 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-tmp-dir\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.630463 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.630443 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-home\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.648585 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.638782 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-model-cache\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.690250 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.690161 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl4ct\" (UniqueName: \"kubernetes.io/projected/7735d239-4062-4651-84be-4ea42357260e-kube-api-access-bl4ct\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.713304 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.713272 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-dshm\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.718569 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.717608 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7735d239-4062-4651-84be-4ea42357260e-tls-certs\") pod \"auth-test-model-870193-kserve-75fbcd7c65-89smn\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.737498 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.737465 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:28.793951 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:28.793912 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:29.051449 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:47:29.051408 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7735d239_4062_4651_84be_4ea42357260e.slice/crio-41a4bce122531e5b7262074d99c4767caaf84e716b705d3aedff215e4c773e67 WatchSource:0}: Error finding container 41a4bce122531e5b7262074d99c4767caaf84e716b705d3aedff215e4c773e67: Status 404 returned error can't find the container with id 41a4bce122531e5b7262074d99c4767caaf84e716b705d3aedff215e4c773e67 Oct 30 19:47:29.053225 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:29.053198 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f27bee-11a8-48c1-bdd9-3f76fafd653e" path="/var/lib/kubelet/pods/77f27bee-11a8-48c1-bdd9-3f76fafd653e/volumes" Oct 30 19:47:29.053658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:29.053633 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn"] Oct 30 19:47:29.914001 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:29.913961 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" event={"ID":"7735d239-4062-4651-84be-4ea42357260e","Type":"ContainerStarted","Data":"d8d2b4a0ed546e00a3d8c449301002765ef07a8d6b98163d2424d0f813f3ba56"} Oct 30 19:47:29.914214 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:29.914026 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" event={"ID":"7735d239-4062-4651-84be-4ea42357260e","Type":"ContainerStarted","Data":"41a4bce122531e5b7262074d99c4767caaf84e716b705d3aedff215e4c773e67"} Oct 30 19:47:29.953345 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:29.953276 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" podStartSLOduration=1.95325505 podStartE2EDuration="1.95325505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:29.949075186 +0000 UTC m=+2603.023530544" watchObservedRunningTime="2026-10-30 19:47:29.95325505 +0000 UTC m=+2603.027710409" Oct 30 19:47:30.497277 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.497237 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj"] Oct 30 19:47:30.508276 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.508234 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h"] Oct 30 19:47:30.508505 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.508472 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" podUID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" containerName="istio-proxy" containerID="cri-o://f106eca16669eaa9a8e6b80c70630c80179e56c6eefdf382c212cb4cd09848c5" gracePeriod=30 Oct 30 19:47:30.508721 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.508700 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.536380 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.535140 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-54086b-openshift-default-dockercfg-ll6pd\"" Oct 30 19:47:30.537545 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.537511 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj"] Oct 30 19:47:30.592933 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.592897 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-token\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.593138 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.592963 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-credential-socket\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.593209 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.593151 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-workload-certs\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.593209 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.593193 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-envoy\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.593341 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.593311 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istiod-ca-cert\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.593386 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.593374 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v774l\" (UniqueName: \"kubernetes.io/projected/ad094cfc-29f5-44cc-82b8-162680c6d3a9-kube-api-access-v774l\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.593554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.593538 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-podinfo\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.593596 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.593583 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-data\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.593776 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.593759 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-workload-socket\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.719043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.712717 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-podinfo\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.719043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.712794 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-data\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.719043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.712956 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-workload-socket\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.719043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.713077 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-token\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.719043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.713148 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-credential-socket\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.719043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.713242 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-workload-certs\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.719043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.713266 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-envoy\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.719043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.713296 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istiod-ca-cert\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.719043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.713319 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v774l\" (UniqueName: \"kubernetes.io/projected/ad094cfc-29f5-44cc-82b8-162680c6d3a9-kube-api-access-v774l\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.719043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.714332 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-workload-socket\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.719043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.714683 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-data\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.719043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.715377 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-workload-certs\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.731050 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.720400 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istiod-ca-cert\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.731050 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.720691 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-credential-socket\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.771147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.767540 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:30.788500 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.788458 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v774l\" (UniqueName: \"kubernetes.io/projected/ad094cfc-29f5-44cc-82b8-162680c6d3a9-kube-api-access-v774l\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.841968 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.841898 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-token\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.852036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.850472 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-envoy\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.852242 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.852133 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-podinfo\") pod \"e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.873044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.863375 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:30.909073 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.906938 5047 patch_prober.go:29] interesting pod/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:47:30.909073 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:30.907082 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" podUID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:47:31.229370 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:47:31.229335 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad094cfc_29f5_44cc_82b8_162680c6d3a9.slice/crio-a2e5d1f3515d0741a020770a65f379f595b8095c1643973fde4a9f03073c8609 WatchSource:0}: Error finding container a2e5d1f3515d0741a020770a65f379f595b8095c1643973fde4a9f03073c8609: Status 404 returned error can't find the container with id a2e5d1f3515d0741a020770a65f379f595b8095c1643973fde4a9f03073c8609 Oct 30 19:47:31.229516 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:31.229492 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj"] Oct 30 19:47:31.233449 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:31.233413 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:47:31.233572 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:31.233494 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:47:31.233572 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:31.233544 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:47:31.944823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:31.944776 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" event={"ID":"ad094cfc-29f5-44cc-82b8-162680c6d3a9","Type":"ContainerStarted","Data":"37a8b3c2cdc3117e3568cf8c2b4d5041461096b417ae2d5356e70d2615c205c9"} Oct 30 19:47:31.945258 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:31.944877 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" event={"ID":"ad094cfc-29f5-44cc-82b8-162680c6d3a9","Type":"ContainerStarted","Data":"a2e5d1f3515d0741a020770a65f379f595b8095c1643973fde4a9f03073c8609"} Oct 30 19:47:32.004042 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:32.003967 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" podStartSLOduration=2.003951734 podStartE2EDuration="2.003951734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:31.991395036 +0000 UTC m=+2605.065850395" watchObservedRunningTime="2026-10-30 19:47:32.003951734 +0000 UTC m=+2605.078407084" Oct 30 19:47:32.864766 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:32.864727 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:32.871607 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:32.871566 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:32.952590 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:32.952546 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:32.953535 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:32.953513 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:47:35.851550 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.851518 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:47:35.949596 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.949547 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-podinfo\") pod \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " Oct 30 19:47:35.949797 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.949622 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-envoy\") pod \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " Oct 30 19:47:35.949797 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.949690 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-workload-socket\") pod \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " Oct 30 19:47:35.949797 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.949766 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-credential-socket\") pod \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " Oct 30 19:47:35.949963 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.949812 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-workload-certs\") pod \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " Oct 30 19:47:35.949963 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.949891 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-data\") pod \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " Oct 30 19:47:35.950647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.950163 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istiod-ca-cert\") pod \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " Oct 30 19:47:35.950647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.950192 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-workload-socket" pod "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" (UID: "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:35.950647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.950220 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-kube-api-access-2rcfn\" (UniqueName: \"kubernetes.io/projected/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-kube-api-access-2rcfn\") pod \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " Oct 30 19:47:35.950647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.950284 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-credential-socket" pod "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" (UID: "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:35.950647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.950295 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-token\" (UniqueName: \"kubernetes.io/projected/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-token\") pod \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\" (UID: \"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c\") " Oct 30 19:47:35.950647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.950432 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-data" pod "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" (UID: "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:35.950647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.950586 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-workload-certs" pod "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" (UID: "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:35.950647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.950612 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istiod-ca-cert" pod "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" (UID: "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:35.952159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.952106 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:35.952403 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.952136 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:35.952403 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.952351 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:35.952403 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.952369 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:35.952403 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.952382 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:35.953076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.953036 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-envoy" pod "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" (UID: "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:35.953525 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.953491 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-kube-api-access-2rcfn" pod "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" (UID: "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c"). InnerVolumeSpecName "kube-api-access-2rcfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:35.953635 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.953580 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-token" pod "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" (UID: "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:35.953885 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.953861 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-podinfo" pod "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" (UID: "96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:47:35.976292 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.976253 5047 generic.go:356] "Generic (PLEG): container finished" podID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" containerID="f106eca16669eaa9a8e6b80c70630c80179e56c6eefdf382c212cb4cd09848c5" exitCode=0 Oct 30 19:47:35.976437 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.976318 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" Oct 30 19:47:35.976437 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.976334 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" event={"ID":"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c","Type":"ContainerDied","Data":"f106eca16669eaa9a8e6b80c70630c80179e56c6eefdf382c212cb4cd09848c5"} Oct 30 19:47:35.976437 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.976404 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h" event={"ID":"96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c","Type":"ContainerDied","Data":"8f40d8a8af589807b83f7f7a8c02f10c6b8f8db7ae2cc46f9b94f7eb3679b8e2"} Oct 30 19:47:35.976437 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.976435 5047 scope.go:122] "RemoveContainer" containerID="f106eca16669eaa9a8e6b80c70630c80179e56c6eefdf382c212cb4cd09848c5" Oct 30 19:47:35.993639 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:35.993606 5047 scope.go:122] "RemoveContainer" containerID="69456a855408608c1513822806b145329f66d796c7f1281804f4aec66378faab" Oct 30 19:47:36.006066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:36.005448 5047 scope.go:122] "RemoveContainer" containerID="f106eca16669eaa9a8e6b80c70630c80179e56c6eefdf382c212cb4cd09848c5" Oct 30 19:47:36.006066 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:47:36.005907 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f106eca16669eaa9a8e6b80c70630c80179e56c6eefdf382c212cb4cd09848c5\": container with ID starting with f106eca16669eaa9a8e6b80c70630c80179e56c6eefdf382c212cb4cd09848c5 not found: ID does not exist" containerID="f106eca16669eaa9a8e6b80c70630c80179e56c6eefdf382c212cb4cd09848c5" Oct 30 19:47:36.006066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:36.005946 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f106eca16669eaa9a8e6b80c70630c80179e56c6eefdf382c212cb4cd09848c5"} err="failed to get container status \"f106eca16669eaa9a8e6b80c70630c80179e56c6eefdf382c212cb4cd09848c5\": rpc error: code = NotFound desc = could not find container \"f106eca16669eaa9a8e6b80c70630c80179e56c6eefdf382c212cb4cd09848c5\": container with ID starting with f106eca16669eaa9a8e6b80c70630c80179e56c6eefdf382c212cb4cd09848c5 not found: ID does not exist" Oct 30 19:47:36.006066 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:36.005971 5047 scope.go:122] "RemoveContainer" containerID="69456a855408608c1513822806b145329f66d796c7f1281804f4aec66378faab" Oct 30 19:47:36.007925 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:47:36.006349 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69456a855408608c1513822806b145329f66d796c7f1281804f4aec66378faab\": container with ID starting with 69456a855408608c1513822806b145329f66d796c7f1281804f4aec66378faab not found: ID does not exist" containerID="69456a855408608c1513822806b145329f66d796c7f1281804f4aec66378faab" Oct 30 19:47:36.007925 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:36.006386 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69456a855408608c1513822806b145329f66d796c7f1281804f4aec66378faab"} err="failed to get container status \"69456a855408608c1513822806b145329f66d796c7f1281804f4aec66378faab\": rpc error: code = NotFound desc = could not find container \"69456a855408608c1513822806b145329f66d796c7f1281804f4aec66378faab\": container with ID starting with 69456a855408608c1513822806b145329f66d796c7f1281804f4aec66378faab not found: ID does not exist" Oct 30 19:47:36.024928 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:36.024891 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h"] Oct 30 19:47:36.031136 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:36.031103 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-08bcfe-openshift-default-6469675667-7b76h"] Oct 30 19:47:36.053720 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:36.053686 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rcfn\" (UniqueName: \"kubernetes.io/projected/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-kube-api-access-2rcfn\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:36.053720 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:36.053717 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:36.053720 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:36.053727 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:36.053953 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:36.053737 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:36.793854 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:36.793820 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:36.806786 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:36.806755 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:36.998193 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:36.998158 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:47:37.055484 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:37.055398 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" path="/var/lib/kubelet/pods/96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c/volumes" Oct 30 19:47:38.737984 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:38.737942 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:38.750855 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:38.750823 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:38.831700 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:38.831630 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" podUID="3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc" containerName="main" containerID="cri-o://590a01d503bcc07f98ac6f81f3e0224c4be8b9d9676dc625155d7465221d9642" gracePeriod=45 Oct 30 19:47:39.009412 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.009373 5047 generic.go:356] "Generic (PLEG): container finished" podID="3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc" containerID="590a01d503bcc07f98ac6f81f3e0224c4be8b9d9676dc625155d7465221d9642" exitCode=0 Oct 30 19:47:39.009536 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.009443 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" event={"ID":"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc","Type":"ContainerDied","Data":"590a01d503bcc07f98ac6f81f3e0224c4be8b9d9676dc625155d7465221d9642"} Oct 30 19:47:39.022796 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.022760 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:47:39.227898 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.227866 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:47:39.306651 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.306538 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-home\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-home\") pod \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " Oct 30 19:47:39.306827 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.306739 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-dshm\") pod \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " Oct 30 19:47:39.306827 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.306807 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-tls-certs\") pod \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " Oct 30 19:47:39.306964 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.306824 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-home" pod "3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc" (UID: "3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:39.306964 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.306842 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-model-cache\") pod \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " Oct 30 19:47:39.306964 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.306919 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-tmp-dir\") pod \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " Oct 30 19:47:39.307193 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.307101 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-kube-api-access-v4k9q\" (UniqueName: \"kubernetes.io/projected/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-kube-api-access-v4k9q\") pod \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\" (UID: \"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc\") " Oct 30 19:47:39.307245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.307183 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-model-cache" pod "3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc" (UID: "3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:39.307389 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.307364 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-tmp-dir" pod "3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc" (UID: "3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:39.307784 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.307753 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:39.307784 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.307779 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:39.307931 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.307792 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:39.310153 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.310086 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-dshm" pod "3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc" (UID: "3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:39.310278 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.310185 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-kube-api-access-v4k9q" pod "3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc" (UID: "3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc"). InnerVolumeSpecName "kube-api-access-v4k9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:39.310278 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.310156 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-tls-certs" pod "3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc" (UID: "3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:39.409141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.409101 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:39.409141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.409135 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:39.409141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:39.409147 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4k9q\" (UniqueName: \"kubernetes.io/projected/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc-kube-api-access-v4k9q\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:47:39.603923 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:47:39.603879 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:47:40.018852 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:40.018758 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" Oct 30 19:47:40.018852 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:40.018779 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" event={"ID":"3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc","Type":"ContainerDied","Data":"8144a61699adc41be7e4a2482d8d1361fd204365c10e873366c752aa7764ebb8"} Oct 30 19:47:40.018852 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:40.018834 5047 scope.go:122] "RemoveContainer" containerID="590a01d503bcc07f98ac6f81f3e0224c4be8b9d9676dc625155d7465221d9642" Oct 30 19:47:40.069160 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:40.069123 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd"] Oct 30 19:47:40.081081 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:40.081042 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd"] Oct 30 19:47:40.219973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:40.219920 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e3504b/test-model-e3504b-kserve-56656d9545-58dxd" podUID="3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.166:8000/ready\": context deadline exceeded" Oct 30 19:47:41.053615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:47:41.053575 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc" path="/var/lib/kubelet/pods/3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc/volumes" Oct 30 19:48:01.166310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:01.166275 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh"] Oct 30 19:48:01.166891 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:01.166507 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" podUID="96bf353d-9dac-4c62-ac9e-c97d305d08a4" containerName="istio-proxy" containerID="cri-o://b209e39dd0bd4727938fd9d590d76eb7b361e874d021f44fe2e239457c2aa2d6" gracePeriod=30 Oct 30 19:48:06.477255 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.477150 5047 generic.go:356] "Generic (PLEG): container finished" podID="96bf353d-9dac-4c62-ac9e-c97d305d08a4" containerID="b209e39dd0bd4727938fd9d590d76eb7b361e874d021f44fe2e239457c2aa2d6" exitCode=0 Oct 30 19:48:06.477756 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.477328 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" event={"ID":"96bf353d-9dac-4c62-ac9e-c97d305d08a4","Type":"ContainerDied","Data":"b209e39dd0bd4727938fd9d590d76eb7b361e874d021f44fe2e239457c2aa2d6"} Oct 30 19:48:06.477756 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.477413 5047 scope.go:122] "RemoveContainer" containerID="f484110505d1ebb6d86ee82833d9148eb10a96b87236e75a4f02ce47931e44de" Oct 30 19:48:06.549556 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.549527 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:48:06.623252 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.623213 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-data\") pod \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " Oct 30 19:48:06.623453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.623363 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-envoy\") pod \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " Oct 30 19:48:06.623453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.623388 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-workload-certs\") pod \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " Oct 30 19:48:06.623453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.623404 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-workload-socket\") pod \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " Oct 30 19:48:06.623645 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.623459 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-podinfo\") pod \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " Oct 30 19:48:06.623645 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.623520 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96bf353d-9dac-4c62-ac9e-c97d305d08a4-kube-api-access-5cfst\" (UniqueName: \"kubernetes.io/projected/96bf353d-9dac-4c62-ac9e-c97d305d08a4-kube-api-access-5cfst\") pod \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " Oct 30 19:48:06.623645 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.623568 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istiod-ca-cert\") pod \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " Oct 30 19:48:06.623645 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.623563 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-data" pod "96bf353d-9dac-4c62-ac9e-c97d305d08a4" (UID: "96bf353d-9dac-4c62-ac9e-c97d305d08a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:06.623645 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.623632 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-credential-socket\") pod \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " Oct 30 19:48:06.623908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.623687 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-token\" (UniqueName: \"kubernetes.io/projected/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-token\") pod \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\" (UID: \"96bf353d-9dac-4c62-ac9e-c97d305d08a4\") " Oct 30 19:48:06.623908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.623823 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-workload-socket" pod "96bf353d-9dac-4c62-ac9e-c97d305d08a4" (UID: "96bf353d-9dac-4c62-ac9e-c97d305d08a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:06.624392 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.624367 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:06.625179 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.625163 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:06.625350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.624671 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-workload-certs" pod "96bf353d-9dac-4c62-ac9e-c97d305d08a4" (UID: "96bf353d-9dac-4c62-ac9e-c97d305d08a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:06.625483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.624943 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-credential-socket" pod "96bf353d-9dac-4c62-ac9e-c97d305d08a4" (UID: "96bf353d-9dac-4c62-ac9e-c97d305d08a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:06.625594 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.625113 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istiod-ca-cert" pod "96bf353d-9dac-4c62-ac9e-c97d305d08a4" (UID: "96bf353d-9dac-4c62-ac9e-c97d305d08a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:06.626296 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.626263 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-token" pod "96bf353d-9dac-4c62-ac9e-c97d305d08a4" (UID: "96bf353d-9dac-4c62-ac9e-c97d305d08a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:06.626386 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.626344 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-envoy" pod "96bf353d-9dac-4c62-ac9e-c97d305d08a4" (UID: "96bf353d-9dac-4c62-ac9e-c97d305d08a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:06.626448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.626401 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96bf353d-9dac-4c62-ac9e-c97d305d08a4-kube-api-access-5cfst" pod "96bf353d-9dac-4c62-ac9e-c97d305d08a4" (UID: "96bf353d-9dac-4c62-ac9e-c97d305d08a4"). InnerVolumeSpecName "kube-api-access-5cfst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:06.627250 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.627223 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-podinfo" pod "96bf353d-9dac-4c62-ac9e-c97d305d08a4" (UID: "96bf353d-9dac-4c62-ac9e-c97d305d08a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:48:06.726988 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.726947 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:06.726988 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.726981 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:06.726988 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.726990 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:06.726988 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.727001 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cfst\" (UniqueName: \"kubernetes.io/projected/96bf353d-9dac-4c62-ac9e-c97d305d08a4-kube-api-access-5cfst\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:06.726988 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.727031 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:06.727357 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.727046 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96bf353d-9dac-4c62-ac9e-c97d305d08a4-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:06.727357 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:06.727055 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96bf353d-9dac-4c62-ac9e-c97d305d08a4-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:07.486985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:07.486947 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" Oct 30 19:48:07.487493 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:07.486990 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh" event={"ID":"96bf353d-9dac-4c62-ac9e-c97d305d08a4","Type":"ContainerDied","Data":"7613c7f9054d8a61dfc56bb6087390c51445a92cace340e964aa6a8439294995"} Oct 30 19:48:07.487493 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:07.487057 5047 scope.go:122] "RemoveContainer" containerID="b209e39dd0bd4727938fd9d590d76eb7b361e874d021f44fe2e239457c2aa2d6" Oct 30 19:48:07.546278 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:07.546242 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh"] Oct 30 19:48:07.549898 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:07.549869 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-614c02-openshift-default-5984c4975-frdxh"] Oct 30 19:48:09.050761 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:09.050726 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96bf353d-9dac-4c62-ac9e-c97d305d08a4" path="/var/lib/kubelet/pods/96bf353d-9dac-4c62-ac9e-c97d305d08a4/volumes" Oct 30 19:48:09.602033 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:48:09.601963 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:48:22.081587 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.081484 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r"] Oct 30 19:48:22.082839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.082812 5047 cpu_manager.go:414] "containerMap: removing container" podUID="96bf353d-9dac-4c62-ac9e-c97d305d08a4" containerName="istio-proxy" Oct 30 19:48:22.082839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.082836 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96bf353d-9dac-4c62-ac9e-c97d305d08a4" containerName="istio-proxy" Oct 30 19:48:22.082979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.082860 5047 cpu_manager.go:414] "containerMap: removing container" podUID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" containerName="istio-proxy" Oct 30 19:48:22.082979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.082866 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" containerName="istio-proxy" Oct 30 19:48:22.082979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.082882 5047 cpu_manager.go:414] "containerMap: removing container" podUID="3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc" containerName="main" Oct 30 19:48:22.082979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.082887 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc" containerName="main" Oct 30 19:48:22.083126 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.082998 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="96bf353d-9dac-4c62-ac9e-c97d305d08a4" containerName="istio-proxy" Oct 30 19:48:22.083126 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.083027 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" containerName="istio-proxy" Oct 30 19:48:22.083126 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.083040 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9e6db1-6c49-4d93-bdfb-85d984c8e9bc" containerName="main" Oct 30 19:48:22.083126 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.083050 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" containerName="istio-proxy" Oct 30 19:48:22.099317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.099286 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.126150 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.126115 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-ec2fe7-openshift-default-dockercfg-c97rm\"" Oct 30 19:48:22.136309 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.136276 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r"] Oct 30 19:48:22.151182 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.151150 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-envoy\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.151333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.151297 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-data\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.151422 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.151381 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-workload-socket\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.151500 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.151481 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxf2x\" (UniqueName: \"kubernetes.io/projected/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-kube-api-access-wxf2x\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.151566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.151549 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-workload-certs\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.151631 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.151590 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istiod-ca-cert\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.151687 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.151626 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-token\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.151687 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.151660 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-podinfo\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.151852 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.151684 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-credential-socket\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.253415 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.253358 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-envoy\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.253605 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.253474 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-data\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.253605 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.253510 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-workload-socket\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.253605 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.253599 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxf2x\" (UniqueName: \"kubernetes.io/projected/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-kube-api-access-wxf2x\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.253800 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.253652 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-workload-certs\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.253800 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.253683 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istiod-ca-cert\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.253800 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.253717 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-token\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.253800 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.253749 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-podinfo\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.253800 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.253772 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-credential-socket\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.254348 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.254318 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-credential-socket\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.254574 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.254555 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-workload-certs\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.255214 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.255188 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istiod-ca-cert\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.255557 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.255532 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-data\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.256045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.256022 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-workload-socket\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.258368 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.258342 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-podinfo\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.259189 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.259148 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-envoy\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.271249 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.271223 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-token\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.271334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.271302 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxf2x\" (UniqueName: \"kubernetes.io/projected/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-kube-api-access-wxf2x\") pod \"e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.438100 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.436977 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:22.955760 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:22.952468 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r"] Oct 30 19:48:23.000946 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:23.000901 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:48:23.001140 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:23.001000 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:48:23.001140 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:23.001062 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:48:23.641537 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:23.641505 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" event={"ID":"ebafb097-ceaa-43ca-9ce4-023cdca5ad56","Type":"ContainerStarted","Data":"953e9794a673ece6ad8731c54373acb7f6362680a74e0be35a4a680dd8b1cdc7"} Oct 30 19:48:23.641923 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:23.641557 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" event={"ID":"ebafb097-ceaa-43ca-9ce4-023cdca5ad56","Type":"ContainerStarted","Data":"5f2638ce77a4e2bc50a52414638618bf6a6ccdf04aadbc7a4605b370ee9ee39b"} Oct 30 19:48:23.702482 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:23.702431 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" podStartSLOduration=1.702416868 podStartE2EDuration="1.702416868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:23.690181102 +0000 UTC m=+2656.764636470" watchObservedRunningTime="2026-10-30 19:48:23.702416868 +0000 UTC m=+2656.776872225" Oct 30 19:48:24.442688 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:24.442650 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:24.449018 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:24.448978 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:24.649516 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:24.649476 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:24.650594 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:24.650570 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:48:25.181612 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:25.181576 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9"] Oct 30 19:48:39.602248 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:48:39.602213 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:48:40.204191 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.204146 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" podUID="b774b412-c5c8-49cc-b504-4b7418fead88" containerName="main" containerID="cri-o://4877dfecfa704e94a76740dcf9636659e465b06179d79bcb14be8d57002733c5" gracePeriod=45 Oct 30 19:48:40.491831 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.491802 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:48:40.651046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.650968 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b774b412-c5c8-49cc-b504-4b7418fead88-tls-certs\" (UniqueName: \"kubernetes.io/secret/b774b412-c5c8-49cc-b504-4b7418fead88-tls-certs\") pod \"b774b412-c5c8-49cc-b504-4b7418fead88\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " Oct 30 19:48:40.651565 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.651061 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-tmp-dir\") pod \"b774b412-c5c8-49cc-b504-4b7418fead88\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " Oct 30 19:48:40.651565 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.651145 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-model-cache\") pod \"b774b412-c5c8-49cc-b504-4b7418fead88\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " Oct 30 19:48:40.651565 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.651231 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b774b412-c5c8-49cc-b504-4b7418fead88-kube-api-access-86qjr\" (UniqueName: \"kubernetes.io/projected/b774b412-c5c8-49cc-b504-4b7418fead88-kube-api-access-86qjr\") pod \"b774b412-c5c8-49cc-b504-4b7418fead88\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " Oct 30 19:48:40.651565 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.651262 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-dshm\") pod \"b774b412-c5c8-49cc-b504-4b7418fead88\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " Oct 30 19:48:40.651565 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.651281 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-tmp-dir" pod "b774b412-c5c8-49cc-b504-4b7418fead88" (UID: "b774b412-c5c8-49cc-b504-4b7418fead88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:40.651565 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.651388 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-home\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-home\") pod \"b774b412-c5c8-49cc-b504-4b7418fead88\" (UID: \"b774b412-c5c8-49cc-b504-4b7418fead88\") " Oct 30 19:48:40.651565 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.651403 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-model-cache" pod "b774b412-c5c8-49cc-b504-4b7418fead88" (UID: "b774b412-c5c8-49cc-b504-4b7418fead88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:40.651918 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.651589 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-home" pod "b774b412-c5c8-49cc-b504-4b7418fead88" (UID: "b774b412-c5c8-49cc-b504-4b7418fead88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:40.652097 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.652077 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:40.652178 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.652123 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:40.652178 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.652138 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:40.653390 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.653367 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b774b412-c5c8-49cc-b504-4b7418fead88-tls-certs" pod "b774b412-c5c8-49cc-b504-4b7418fead88" (UID: "b774b412-c5c8-49cc-b504-4b7418fead88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:40.653474 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.653408 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-dshm" pod "b774b412-c5c8-49cc-b504-4b7418fead88" (UID: "b774b412-c5c8-49cc-b504-4b7418fead88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:40.654575 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.654554 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b774b412-c5c8-49cc-b504-4b7418fead88-kube-api-access-86qjr" pod "b774b412-c5c8-49cc-b504-4b7418fead88" (UID: "b774b412-c5c8-49cc-b504-4b7418fead88"). InnerVolumeSpecName "kube-api-access-86qjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:40.753319 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.753234 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86qjr\" (UniqueName: \"kubernetes.io/projected/b774b412-c5c8-49cc-b504-4b7418fead88-kube-api-access-86qjr\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:40.753319 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.753266 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b774b412-c5c8-49cc-b504-4b7418fead88-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:40.753319 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.753279 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b774b412-c5c8-49cc-b504-4b7418fead88-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:40.780780 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.780746 5047 generic.go:356] "Generic (PLEG): container finished" podID="b774b412-c5c8-49cc-b504-4b7418fead88" containerID="4877dfecfa704e94a76740dcf9636659e465b06179d79bcb14be8d57002733c5" exitCode=0 Oct 30 19:48:40.780946 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.780794 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" Oct 30 19:48:40.780946 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.780808 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" event={"ID":"b774b412-c5c8-49cc-b504-4b7418fead88","Type":"ContainerDied","Data":"4877dfecfa704e94a76740dcf9636659e465b06179d79bcb14be8d57002733c5"} Oct 30 19:48:40.780946 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.780845 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9" event={"ID":"b774b412-c5c8-49cc-b504-4b7418fead88","Type":"ContainerDied","Data":"2acd6497fef8a2c1980a9340c04d72fa25137e8121c9bc601bc6d8ed6dfabfb8"} Oct 30 19:48:40.780946 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.780861 5047 scope.go:122] "RemoveContainer" containerID="4877dfecfa704e94a76740dcf9636659e465b06179d79bcb14be8d57002733c5" Oct 30 19:48:40.795908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.795866 5047 scope.go:122] "RemoveContainer" containerID="4877dfecfa704e94a76740dcf9636659e465b06179d79bcb14be8d57002733c5" Oct 30 19:48:40.796416 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:48:40.796389 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4877dfecfa704e94a76740dcf9636659e465b06179d79bcb14be8d57002733c5\": container with ID starting with 4877dfecfa704e94a76740dcf9636659e465b06179d79bcb14be8d57002733c5 not found: ID does not exist" containerID="4877dfecfa704e94a76740dcf9636659e465b06179d79bcb14be8d57002733c5" Oct 30 19:48:40.796547 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.796426 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4877dfecfa704e94a76740dcf9636659e465b06179d79bcb14be8d57002733c5"} err="failed to get container status \"4877dfecfa704e94a76740dcf9636659e465b06179d79bcb14be8d57002733c5\": rpc error: code = NotFound desc = could not find container \"4877dfecfa704e94a76740dcf9636659e465b06179d79bcb14be8d57002733c5\": container with ID starting with 4877dfecfa704e94a76740dcf9636659e465b06179d79bcb14be8d57002733c5 not found: ID does not exist" Oct 30 19:48:40.828881 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.828853 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9"] Oct 30 19:48:40.837731 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:40.837707 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1f6cef/test-model-1f6cef-kserve-5dc44d5c4f-zb4m9"] Oct 30 19:48:41.051501 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:41.051421 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b774b412-c5c8-49cc-b504-4b7418fead88" path="/var/lib/kubelet/pods/b774b412-c5c8-49cc-b504-4b7418fead88/volumes" Oct 30 19:48:45.182505 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.182468 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz"] Oct 30 19:48:45.182988 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.182725 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" podUID="33e689ed-675a-4ff5-a379-e2d7e751eb78" containerName="maas-api" containerID="cri-o://a3873884bdfed552f9c326ea36015aa0511f27cc3bf1c0e7c8e11c72a1c9d2c3" gracePeriod=30 Oct 30 19:48:45.718030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.714515 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:48:45.810045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.809944 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e689ed-675a-4ff5-a379-e2d7e751eb78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e689ed-675a-4ff5-a379-e2d7e751eb78-metrics-certs\") pod \"33e689ed-675a-4ff5-a379-e2d7e751eb78\" (UID: \"33e689ed-675a-4ff5-a379-e2d7e751eb78\") " Oct 30 19:48:45.810211 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.810111 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e689ed-675a-4ff5-a379-e2d7e751eb78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e689ed-675a-4ff5-a379-e2d7e751eb78-openshift-service-ca\") pod \"33e689ed-675a-4ff5-a379-e2d7e751eb78\" (UID: \"33e689ed-675a-4ff5-a379-e2d7e751eb78\") " Oct 30 19:48:45.810211 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.810151 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e689ed-675a-4ff5-a379-e2d7e751eb78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e689ed-675a-4ff5-a379-e2d7e751eb78-maas-api-tls\") pod \"33e689ed-675a-4ff5-a379-e2d7e751eb78\" (UID: \"33e689ed-675a-4ff5-a379-e2d7e751eb78\") " Oct 30 19:48:45.810293 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.810223 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e689ed-675a-4ff5-a379-e2d7e751eb78-kube-api-access-f9qxk\" (UniqueName: \"kubernetes.io/projected/33e689ed-675a-4ff5-a379-e2d7e751eb78-kube-api-access-f9qxk\") pod \"33e689ed-675a-4ff5-a379-e2d7e751eb78\" (UID: \"33e689ed-675a-4ff5-a379-e2d7e751eb78\") " Oct 30 19:48:45.810560 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.810528 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e689ed-675a-4ff5-a379-e2d7e751eb78-openshift-service-ca" pod "33e689ed-675a-4ff5-a379-e2d7e751eb78" (UID: "33e689ed-675a-4ff5-a379-e2d7e751eb78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:45.811143 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.811115 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e689ed-675a-4ff5-a379-e2d7e751eb78-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:45.813256 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.813195 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e689ed-675a-4ff5-a379-e2d7e751eb78-maas-api-tls" pod "33e689ed-675a-4ff5-a379-e2d7e751eb78" (UID: "33e689ed-675a-4ff5-a379-e2d7e751eb78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:45.813256 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.813241 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e689ed-675a-4ff5-a379-e2d7e751eb78-metrics-certs" pod "33e689ed-675a-4ff5-a379-e2d7e751eb78" (UID: "33e689ed-675a-4ff5-a379-e2d7e751eb78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:45.813561 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.813537 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e689ed-675a-4ff5-a379-e2d7e751eb78-kube-api-access-f9qxk" pod "33e689ed-675a-4ff5-a379-e2d7e751eb78" (UID: "33e689ed-675a-4ff5-a379-e2d7e751eb78"). InnerVolumeSpecName "kube-api-access-f9qxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:45.825769 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.825538 5047 generic.go:356] "Generic (PLEG): container finished" podID="33e689ed-675a-4ff5-a379-e2d7e751eb78" containerID="a3873884bdfed552f9c326ea36015aa0511f27cc3bf1c0e7c8e11c72a1c9d2c3" exitCode=0 Oct 30 19:48:45.825769 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.825600 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" Oct 30 19:48:45.825769 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.825633 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" event={"ID":"33e689ed-675a-4ff5-a379-e2d7e751eb78","Type":"ContainerDied","Data":"a3873884bdfed552f9c326ea36015aa0511f27cc3bf1c0e7c8e11c72a1c9d2c3"} Oct 30 19:48:45.826018 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.825724 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz" event={"ID":"33e689ed-675a-4ff5-a379-e2d7e751eb78","Type":"ContainerDied","Data":"8e1789ce5274f8f5f8ef2f7b1d2df204a286554675583a61749a6f1988a41031"} Oct 30 19:48:45.826018 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.825764 5047 scope.go:122] "RemoveContainer" containerID="a3873884bdfed552f9c326ea36015aa0511f27cc3bf1c0e7c8e11c72a1c9d2c3" Oct 30 19:48:45.850550 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.850527 5047 scope.go:122] "RemoveContainer" containerID="a3873884bdfed552f9c326ea36015aa0511f27cc3bf1c0e7c8e11c72a1c9d2c3" Oct 30 19:48:45.850908 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:48:45.850885 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3873884bdfed552f9c326ea36015aa0511f27cc3bf1c0e7c8e11c72a1c9d2c3\": container with ID starting with a3873884bdfed552f9c326ea36015aa0511f27cc3bf1c0e7c8e11c72a1c9d2c3 not found: ID does not exist" containerID="a3873884bdfed552f9c326ea36015aa0511f27cc3bf1c0e7c8e11c72a1c9d2c3" Oct 30 19:48:45.851021 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.850918 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3873884bdfed552f9c326ea36015aa0511f27cc3bf1c0e7c8e11c72a1c9d2c3"} err="failed to get container status \"a3873884bdfed552f9c326ea36015aa0511f27cc3bf1c0e7c8e11c72a1c9d2c3\": rpc error: code = NotFound desc = could not find container \"a3873884bdfed552f9c326ea36015aa0511f27cc3bf1c0e7c8e11c72a1c9d2c3\": container with ID starting with a3873884bdfed552f9c326ea36015aa0511f27cc3bf1c0e7c8e11c72a1c9d2c3 not found: ID does not exist" Oct 30 19:48:45.879631 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.879596 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz"] Oct 30 19:48:45.885674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.885644 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3504b-7db5f8c89-kg5wz"] Oct 30 19:48:45.912803 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.912776 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e689ed-675a-4ff5-a379-e2d7e751eb78-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:45.912803 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.912802 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9qxk\" (UniqueName: \"kubernetes.io/projected/33e689ed-675a-4ff5-a379-e2d7e751eb78-kube-api-access-f9qxk\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:45.912992 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:45.912812 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e689ed-675a-4ff5-a379-e2d7e751eb78-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:48:47.052088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:48:47.052055 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e689ed-675a-4ff5-a379-e2d7e751eb78" path="/var/lib/kubelet/pods/33e689ed-675a-4ff5-a379-e2d7e751eb78/volumes" Oct 30 19:49:01.634666 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.634624 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b"] Oct 30 19:49:01.635862 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.635836 5047 cpu_manager.go:414] "containerMap: removing container" podUID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" containerName="istio-proxy" Oct 30 19:49:01.635957 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.635862 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96dfcc8c-fce0-4ed3-8fcd-e30c4d67940c" containerName="istio-proxy" Oct 30 19:49:01.635957 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.635879 5047 cpu_manager.go:414] "containerMap: removing container" podUID="b774b412-c5c8-49cc-b504-4b7418fead88" containerName="main" Oct 30 19:49:01.635957 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.635886 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b774b412-c5c8-49cc-b504-4b7418fead88" containerName="main" Oct 30 19:49:01.636142 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.635974 5047 cpu_manager.go:414] "containerMap: removing container" podUID="33e689ed-675a-4ff5-a379-e2d7e751eb78" containerName="maas-api" Oct 30 19:49:01.636142 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.635983 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e689ed-675a-4ff5-a379-e2d7e751eb78" containerName="maas-api" Oct 30 19:49:01.636142 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.636039 5047 cpu_manager.go:414] "containerMap: removing container" podUID="96bf353d-9dac-4c62-ac9e-c97d305d08a4" containerName="istio-proxy" Oct 30 19:49:01.636142 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.636048 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96bf353d-9dac-4c62-ac9e-c97d305d08a4" containerName="istio-proxy" Oct 30 19:49:01.636344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.636247 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e689ed-675a-4ff5-a379-e2d7e751eb78" containerName="maas-api" Oct 30 19:49:01.636344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.636263 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="96bf353d-9dac-4c62-ac9e-c97d305d08a4" containerName="istio-proxy" Oct 30 19:49:01.636344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.636277 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="b774b412-c5c8-49cc-b504-4b7418fead88" containerName="main" Oct 30 19:49:01.641764 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.641736 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 19:49:01.662998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.662961 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b"] Oct 30 19:49:01.663694 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.663652 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d40dfe2-d6ef-449d-a678-8e6938ea132e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b\" (UID: \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 19:49:01.663813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.663771 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d40dfe2-d6ef-449d-a678-8e6938ea132e-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b\" (UID: \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 19:49:01.663866 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.663816 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d40dfe2-d6ef-449d-a678-8e6938ea132e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b\" (UID: \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 19:49:01.663987 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.663970 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlsr5\" (UniqueName: \"kubernetes.io/projected/3d40dfe2-d6ef-449d-a678-8e6938ea132e-kube-api-access-rlsr5\") pod \"maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b\" (UID: \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 19:49:01.666592 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.666549 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-ec2fe7-serving-cert\"" Oct 30 19:49:01.765899 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.765859 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlsr5\" (UniqueName: \"kubernetes.io/projected/3d40dfe2-d6ef-449d-a678-8e6938ea132e-kube-api-access-rlsr5\") pod \"maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b\" (UID: \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 19:49:01.766156 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.765975 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d40dfe2-d6ef-449d-a678-8e6938ea132e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b\" (UID: \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 19:49:01.766156 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.766129 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d40dfe2-d6ef-449d-a678-8e6938ea132e-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b\" (UID: \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 19:49:01.766275 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.766174 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d40dfe2-d6ef-449d-a678-8e6938ea132e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b\" (UID: \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 19:49:01.766820 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.766796 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d40dfe2-d6ef-449d-a678-8e6938ea132e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b\" (UID: \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 19:49:01.768644 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.768626 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d40dfe2-d6ef-449d-a678-8e6938ea132e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b\" (UID: \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 19:49:01.768955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.768940 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d40dfe2-d6ef-449d-a678-8e6938ea132e-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b\" (UID: \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 19:49:01.783672 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.783649 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlsr5\" (UniqueName: \"kubernetes.io/projected/3d40dfe2-d6ef-449d-a678-8e6938ea132e-kube-api-access-rlsr5\") pod \"maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b\" (UID: \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 19:49:01.963318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:01.963227 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 19:49:02.283308 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:02.283272 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b"] Oct 30 19:49:02.283971 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:49:02.283940 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d40dfe2_d6ef_449d_a678_8e6938ea132e.slice/crio-bfb0e4858b73b9583184fb00379a44d03c98b9c1df87f296756ec40721ff1f00 WatchSource:0}: Error finding container bfb0e4858b73b9583184fb00379a44d03c98b9c1df87f296756ec40721ff1f00: Status 404 returned error can't find the container with id bfb0e4858b73b9583184fb00379a44d03c98b9c1df87f296756ec40721ff1f00 Oct 30 19:49:03.152697 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:03.152654 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" event={"ID":"3d40dfe2-d6ef-449d-a678-8e6938ea132e","Type":"ContainerStarted","Data":"39d8f08885c12974d504ebe9383d76d73d8363fdec4e0442b6c9df80f6bf1a25"} Oct 30 19:49:03.153108 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:03.152723 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr"] Oct 30 19:49:03.153108 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:03.152768 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 19:49:03.153108 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:03.152790 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz"] Oct 30 19:49:03.153108 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:03.152818 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" event={"ID":"3d40dfe2-d6ef-449d-a678-8e6938ea132e","Type":"ContainerStarted","Data":"bfb0e4858b73b9583184fb00379a44d03c98b9c1df87f296756ec40721ff1f00"} Oct 30 19:49:03.165769 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:03.165701 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" podStartSLOduration=1.6894122839999999 podStartE2EDuration="2.165683646s" totalImagesPullingTime="476.271362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:03.159835927 +0000 UTC m=+2696.234291277" watchObservedRunningTime="2026-10-30 19:49:03.165683646 +0000 UTC m=+2696.240139006" Oct 30 19:49:03.656359 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:03.656318 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj"] Oct 30 19:49:03.657355 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:03.657312 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" podUID="299d2abb-3201-42ac-8f33-edc45040171a" containerName="istio-proxy" containerID="cri-o://dcf227a913476a447a2d180c56fb0a4783471f0579bb2fab430cbadf9ca39fe2" gracePeriod=30 Oct 30 19:49:04.999935 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:04.999888 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5"] Oct 30 19:49:05.119237 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.119194 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.165030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.157585 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-5250ee-openshift-default-dockercfg-bzstj\"" Oct 30 19:49:05.229060 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.228887 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-data\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.229287 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.229152 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-podinfo\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.229392 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.229294 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-workload-certs\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.229854 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.229828 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-envoy\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.229948 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.229926 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-workload-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.230344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.230309 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-credential-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.230439 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.230410 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pdzn\" (UniqueName: \"kubernetes.io/projected/9a300de1-9957-43a4-a24c-a7186fe1ba64-kube-api-access-5pdzn\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.230582 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.230563 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-token\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.230638 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.230610 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a300de1-9957-43a4-a24c-a7186fe1ba64-istiod-ca-cert\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.239392 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.239362 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5"] Oct 30 19:49:05.331995 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.331955 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-workload-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.332229 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.332118 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-credential-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.332308 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.332223 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pdzn\" (UniqueName: \"kubernetes.io/projected/9a300de1-9957-43a4-a24c-a7186fe1ba64-kube-api-access-5pdzn\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.332409 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.332391 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-token\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.332524 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.332429 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-workload-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.332524 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.332434 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a300de1-9957-43a4-a24c-a7186fe1ba64-istiod-ca-cert\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.332696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.332557 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-data\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.332696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.332632 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-podinfo\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.332696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.332639 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-credential-socket\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.332696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.332680 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-workload-certs\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.332945 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.332774 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-envoy\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.332945 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.332909 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-data\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.333236 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.333212 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-workload-certs\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.333343 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.333317 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a300de1-9957-43a4-a24c-a7186fe1ba64-istiod-ca-cert\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.335675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.335650 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-envoy\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.336186 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.336162 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-podinfo\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.353726 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.353689 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pdzn\" (UniqueName: \"kubernetes.io/projected/9a300de1-9957-43a4-a24c-a7186fe1ba64-kube-api-access-5pdzn\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.357498 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.357472 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-token\") pod \"e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:05.488124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:05.488088 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:06.076099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:06.076069 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5"] Oct 30 19:49:06.076715 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:49:06.076691 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a300de1_9957_43a4_a24c_a7186fe1ba64.slice/crio-f70cbf2bde7642b95e5803b60e2a25d52c0997d369e17bad759c0e4407d154d5 WatchSource:0}: Error finding container f70cbf2bde7642b95e5803b60e2a25d52c0997d369e17bad759c0e4407d154d5: Status 404 returned error can't find the container with id f70cbf2bde7642b95e5803b60e2a25d52c0997d369e17bad759c0e4407d154d5 Oct 30 19:49:06.079584 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:06.079553 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:06.079661 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:06.079617 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:06.079661 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:06.079642 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:06.258258 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:06.258221 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" event={"ID":"9a300de1-9957-43a4-a24c-a7186fe1ba64","Type":"ContainerStarted","Data":"196eef2e0a36f718f9ad15e2d0c83509829c6f11ed1c90ce834239f8fbfb797f"} Oct 30 19:49:06.258433 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:06.258282 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" event={"ID":"9a300de1-9957-43a4-a24c-a7186fe1ba64","Type":"ContainerStarted","Data":"f70cbf2bde7642b95e5803b60e2a25d52c0997d369e17bad759c0e4407d154d5"} Oct 30 19:49:06.298998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:06.298950 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" podStartSLOduration=2.298934578 podStartE2EDuration="2.298934578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:06.297572461 +0000 UTC m=+2699.372027818" watchObservedRunningTime="2026-10-30 19:49:06.298934578 +0000 UTC m=+2699.373389935" Oct 30 19:49:06.491334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:06.491250 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:07.496697 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:07.496658 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:07.850210 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:07.850178 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:49:07.853256 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:07.853230 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:49:08.153896 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:08.153802 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" podUID="bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5" containerName="payload-processing" containerID="cri-o://5d804544b322c0ed609cd511ad0520c8dbbec82473205a6fa60b81dfcfe63384" gracePeriod=25 Oct 30 19:49:08.154089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:08.153809 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" podUID="313dd358-14e7-4172-b9cb-3594dc0f4e7f" containerName="payload-pre-processing" containerID="cri-o://66c087809cb66c6882225218b8844fbb926c373583222388af34933bb7b2c072" gracePeriod=25 Oct 30 19:49:08.272221 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:08.272158 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:08.273118 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:08.273095 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:49:08.960714 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:08.960684 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:49:09.021611 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.021559 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299d2abb-3201-42ac-8f33-edc45040171a-kube-api-access-rbfkf\" (UniqueName: \"kubernetes.io/projected/299d2abb-3201-42ac-8f33-edc45040171a-kube-api-access-rbfkf\") pod \"299d2abb-3201-42ac-8f33-edc45040171a\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " Oct 30 19:49:09.021809 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.021659 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299d2abb-3201-42ac-8f33-edc45040171a-istio-token\" (UniqueName: \"kubernetes.io/projected/299d2abb-3201-42ac-8f33-edc45040171a-istio-token\") pod \"299d2abb-3201-42ac-8f33-edc45040171a\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " Oct 30 19:49:09.021809 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.021748 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-istio-envoy\") pod \"299d2abb-3201-42ac-8f33-edc45040171a\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " Oct 30 19:49:09.021976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.021814 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-workload-certs\") pod \"299d2abb-3201-42ac-8f33-edc45040171a\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " Oct 30 19:49:09.021976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.021868 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-workload-socket\") pod \"299d2abb-3201-42ac-8f33-edc45040171a\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " Oct 30 19:49:09.022149 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.021978 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-istio-data\") pod \"299d2abb-3201-42ac-8f33-edc45040171a\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " Oct 30 19:49:09.022149 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.022023 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/299d2abb-3201-42ac-8f33-edc45040171a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299d2abb-3201-42ac-8f33-edc45040171a-istiod-ca-cert\") pod \"299d2abb-3201-42ac-8f33-edc45040171a\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " Oct 30 19:49:09.022149 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.022133 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/299d2abb-3201-42ac-8f33-edc45040171a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299d2abb-3201-42ac-8f33-edc45040171a-istio-podinfo\") pod \"299d2abb-3201-42ac-8f33-edc45040171a\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " Oct 30 19:49:09.022381 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.022192 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-credential-socket\") pod \"299d2abb-3201-42ac-8f33-edc45040171a\" (UID: \"299d2abb-3201-42ac-8f33-edc45040171a\") " Oct 30 19:49:09.023224 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.023172 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-workload-certs" pod "299d2abb-3201-42ac-8f33-edc45040171a" (UID: "299d2abb-3201-42ac-8f33-edc45040171a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:09.023352 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.023265 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-credential-socket" pod "299d2abb-3201-42ac-8f33-edc45040171a" (UID: "299d2abb-3201-42ac-8f33-edc45040171a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:09.023880 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.023647 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-workload-socket" pod "299d2abb-3201-42ac-8f33-edc45040171a" (UID: "299d2abb-3201-42ac-8f33-edc45040171a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:09.023880 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.023711 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/299d2abb-3201-42ac-8f33-edc45040171a-istiod-ca-cert" pod "299d2abb-3201-42ac-8f33-edc45040171a" (UID: "299d2abb-3201-42ac-8f33-edc45040171a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:09.023880 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.023851 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-istio-data" pod "299d2abb-3201-42ac-8f33-edc45040171a" (UID: "299d2abb-3201-42ac-8f33-edc45040171a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:09.027136 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.026995 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299d2abb-3201-42ac-8f33-edc45040171a-kube-api-access-rbfkf" pod "299d2abb-3201-42ac-8f33-edc45040171a" (UID: "299d2abb-3201-42ac-8f33-edc45040171a"). InnerVolumeSpecName "kube-api-access-rbfkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:09.028442 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.028413 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299d2abb-3201-42ac-8f33-edc45040171a-istio-token" pod "299d2abb-3201-42ac-8f33-edc45040171a" (UID: "299d2abb-3201-42ac-8f33-edc45040171a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:09.049442 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.049397 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-istio-envoy" pod "299d2abb-3201-42ac-8f33-edc45040171a" (UID: "299d2abb-3201-42ac-8f33-edc45040171a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:09.049551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.049498 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/299d2abb-3201-42ac-8f33-edc45040171a-istio-podinfo" pod "299d2abb-3201-42ac-8f33-edc45040171a" (UID: "299d2abb-3201-42ac-8f33-edc45040171a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:49:09.124536 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.124505 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:09.124689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.124551 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:09.124689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.124576 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299d2abb-3201-42ac-8f33-edc45040171a-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:09.124689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.124592 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299d2abb-3201-42ac-8f33-edc45040171a-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:09.124689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.124607 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:09.124689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.124621 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbfkf\" (UniqueName: \"kubernetes.io/projected/299d2abb-3201-42ac-8f33-edc45040171a-kube-api-access-rbfkf\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:09.124689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.124634 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/299d2abb-3201-42ac-8f33-edc45040171a-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:09.124689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.124647 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:09.124689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.124656 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299d2abb-3201-42ac-8f33-edc45040171a-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:09.134602 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.134580 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 19:49:09.296081 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.290525 5047 generic.go:356] "Generic (PLEG): container finished" podID="299d2abb-3201-42ac-8f33-edc45040171a" containerID="dcf227a913476a447a2d180c56fb0a4783471f0579bb2fab430cbadf9ca39fe2" exitCode=0 Oct 30 19:49:09.296081 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.290701 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" Oct 30 19:49:09.296081 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.290679 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" event={"ID":"299d2abb-3201-42ac-8f33-edc45040171a","Type":"ContainerDied","Data":"dcf227a913476a447a2d180c56fb0a4783471f0579bb2fab430cbadf9ca39fe2"} Oct 30 19:49:09.296081 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.290954 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj" event={"ID":"299d2abb-3201-42ac-8f33-edc45040171a","Type":"ContainerDied","Data":"8b178a3545fe843594f4967e589912d0245939febdb68981e5772aa82c21d734"} Oct 30 19:49:09.296081 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.291018 5047 scope.go:122] "RemoveContainer" containerID="dcf227a913476a447a2d180c56fb0a4783471f0579bb2fab430cbadf9ca39fe2" Oct 30 19:49:09.310751 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.310732 5047 scope.go:122] "RemoveContainer" containerID="000c8cc05ac04716cd74114a6bc3f4d2330f997caed5e8a45280bfb3001cbdb4" Oct 30 19:49:09.320829 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.320808 5047 scope.go:122] "RemoveContainer" containerID="dcf227a913476a447a2d180c56fb0a4783471f0579bb2fab430cbadf9ca39fe2" Oct 30 19:49:09.321141 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:49:09.321115 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf227a913476a447a2d180c56fb0a4783471f0579bb2fab430cbadf9ca39fe2\": container with ID starting with dcf227a913476a447a2d180c56fb0a4783471f0579bb2fab430cbadf9ca39fe2 not found: ID does not exist" containerID="dcf227a913476a447a2d180c56fb0a4783471f0579bb2fab430cbadf9ca39fe2" Oct 30 19:49:09.321224 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.321150 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf227a913476a447a2d180c56fb0a4783471f0579bb2fab430cbadf9ca39fe2"} err="failed to get container status \"dcf227a913476a447a2d180c56fb0a4783471f0579bb2fab430cbadf9ca39fe2\": rpc error: code = NotFound desc = could not find container \"dcf227a913476a447a2d180c56fb0a4783471f0579bb2fab430cbadf9ca39fe2\": container with ID starting with dcf227a913476a447a2d180c56fb0a4783471f0579bb2fab430cbadf9ca39fe2 not found: ID does not exist" Oct 30 19:49:09.321224 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.321168 5047 scope.go:122] "RemoveContainer" containerID="000c8cc05ac04716cd74114a6bc3f4d2330f997caed5e8a45280bfb3001cbdb4" Oct 30 19:49:09.321423 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:49:09.321404 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"000c8cc05ac04716cd74114a6bc3f4d2330f997caed5e8a45280bfb3001cbdb4\": container with ID starting with 000c8cc05ac04716cd74114a6bc3f4d2330f997caed5e8a45280bfb3001cbdb4 not found: ID does not exist" containerID="000c8cc05ac04716cd74114a6bc3f4d2330f997caed5e8a45280bfb3001cbdb4" Oct 30 19:49:09.321481 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.321433 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"000c8cc05ac04716cd74114a6bc3f4d2330f997caed5e8a45280bfb3001cbdb4"} err="failed to get container status \"000c8cc05ac04716cd74114a6bc3f4d2330f997caed5e8a45280bfb3001cbdb4\": rpc error: code = NotFound desc = could not find container \"000c8cc05ac04716cd74114a6bc3f4d2330f997caed5e8a45280bfb3001cbdb4\": container with ID starting with 000c8cc05ac04716cd74114a6bc3f4d2330f997caed5e8a45280bfb3001cbdb4 not found: ID does not exist" Oct 30 19:49:09.341496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.341466 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj"] Oct 30 19:49:09.345832 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:09.345806 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e3504b-openshift-default-668d85bd4c-cbfdj"] Oct 30 19:49:09.602848 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:49:09.602812 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:49:11.055942 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:11.055896 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="299d2abb-3201-42ac-8f33-edc45040171a" path="/var/lib/kubelet/pods/299d2abb-3201-42ac-8f33-edc45040171a/volumes" Oct 30 19:49:12.757886 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:12.757850 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" podUID="313dd358-14e7-4172-b9cb-3594dc0f4e7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:49:12.758280 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:12.757868 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" podUID="bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:49:13.063267 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:13.063082 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz"] Oct 30 19:49:13.064235 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:13.064204 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" podUID="2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" containerName="istio-proxy" containerID="cri-o://80da37ab3e7a5b1fadd5702fc43322124898f3a08b717daba180eacc67b58932" gracePeriod=30 Oct 30 19:49:16.075886 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:16.075848 5047 patch_prober.go:29] interesting pod/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:49:16.076304 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:16.075916 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" podUID="2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:49:18.363083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.363055 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:49:18.382836 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.382802 5047 generic.go:356] "Generic (PLEG): container finished" podID="2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" containerID="80da37ab3e7a5b1fadd5702fc43322124898f3a08b717daba180eacc67b58932" exitCode=0 Oct 30 19:49:18.382992 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.382857 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" Oct 30 19:49:18.382992 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.382889 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" event={"ID":"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760","Type":"ContainerDied","Data":"80da37ab3e7a5b1fadd5702fc43322124898f3a08b717daba180eacc67b58932"} Oct 30 19:49:18.382992 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.382958 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz" event={"ID":"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760","Type":"ContainerDied","Data":"2eba252084590efbf9ec46495c0b371c8ecbdffa69b52b7a18fa8b79c2d89a7c"} Oct 30 19:49:18.383142 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.383017 5047 scope.go:122] "RemoveContainer" containerID="80da37ab3e7a5b1fadd5702fc43322124898f3a08b717daba180eacc67b58932" Oct 30 19:49:18.394144 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.394126 5047 scope.go:122] "RemoveContainer" containerID="927e7a7ecdb4a82d5cb71d61feaf75a51eda445e695fd3b3d6e59fdd033f2782" Oct 30 19:49:18.407165 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.407137 5047 scope.go:122] "RemoveContainer" containerID="80da37ab3e7a5b1fadd5702fc43322124898f3a08b717daba180eacc67b58932" Oct 30 19:49:18.407622 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:49:18.407596 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80da37ab3e7a5b1fadd5702fc43322124898f3a08b717daba180eacc67b58932\": container with ID starting with 80da37ab3e7a5b1fadd5702fc43322124898f3a08b717daba180eacc67b58932 not found: ID does not exist" containerID="80da37ab3e7a5b1fadd5702fc43322124898f3a08b717daba180eacc67b58932" Oct 30 19:49:18.407678 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.407636 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80da37ab3e7a5b1fadd5702fc43322124898f3a08b717daba180eacc67b58932"} err="failed to get container status \"80da37ab3e7a5b1fadd5702fc43322124898f3a08b717daba180eacc67b58932\": rpc error: code = NotFound desc = could not find container \"80da37ab3e7a5b1fadd5702fc43322124898f3a08b717daba180eacc67b58932\": container with ID starting with 80da37ab3e7a5b1fadd5702fc43322124898f3a08b717daba180eacc67b58932 not found: ID does not exist" Oct 30 19:49:18.407678 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.407659 5047 scope.go:122] "RemoveContainer" containerID="927e7a7ecdb4a82d5cb71d61feaf75a51eda445e695fd3b3d6e59fdd033f2782" Oct 30 19:49:18.407951 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:49:18.407926 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"927e7a7ecdb4a82d5cb71d61feaf75a51eda445e695fd3b3d6e59fdd033f2782\": container with ID starting with 927e7a7ecdb4a82d5cb71d61feaf75a51eda445e695fd3b3d6e59fdd033f2782 not found: ID does not exist" containerID="927e7a7ecdb4a82d5cb71d61feaf75a51eda445e695fd3b3d6e59fdd033f2782" Oct 30 19:49:18.408071 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.407961 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"927e7a7ecdb4a82d5cb71d61feaf75a51eda445e695fd3b3d6e59fdd033f2782"} err="failed to get container status \"927e7a7ecdb4a82d5cb71d61feaf75a51eda445e695fd3b3d6e59fdd033f2782\": rpc error: code = NotFound desc = could not find container \"927e7a7ecdb4a82d5cb71d61feaf75a51eda445e695fd3b3d6e59fdd033f2782\": container with ID starting with 927e7a7ecdb4a82d5cb71d61feaf75a51eda445e695fd3b3d6e59fdd033f2782 not found: ID does not exist" Oct 30 19:49:18.431069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.431040 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-workload-certs\") pod \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " Oct 30 19:49:18.431212 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.431083 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istiod-ca-cert\") pod \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " Oct 30 19:49:18.431212 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.431118 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-envoy\") pod \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " Oct 30 19:49:18.431212 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.431182 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-podinfo\") pod \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " Oct 30 19:49:18.431335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.431257 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-data\") pod \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " Oct 30 19:49:18.431335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.431303 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-credential-socket\") pod \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " Oct 30 19:49:18.431335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.431313 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-workload-certs" pod "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" (UID: "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:18.431444 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.431423 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-token\" (UniqueName: \"kubernetes.io/projected/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-token\") pod \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " Oct 30 19:49:18.431485 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.431454 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-workload-socket\") pod \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " Oct 30 19:49:18.431545 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.431518 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-kube-api-access-dh6td\" (UniqueName: \"kubernetes.io/projected/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-kube-api-access-dh6td\") pod \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\" (UID: \"2d8f6efb-ac1d-4d9a-9a19-f85f09e01760\") " Oct 30 19:49:18.431545 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.431528 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-credential-socket" pod "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" (UID: "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:18.431832 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.431809 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istiod-ca-cert" pod "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" (UID: "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:18.432188 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.432164 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-data" pod "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" (UID: "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:18.432538 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.432518 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:18.432538 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.432522 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-workload-socket" pod "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" (UID: "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:18.432679 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.432541 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:18.432679 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.432565 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:18.432679 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.432589 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:18.434691 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.434665 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-token" pod "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" (UID: "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:18.434872 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.434852 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-podinfo" pod "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" (UID: "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:49:18.435130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.435106 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-envoy" pod "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" (UID: "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:18.435793 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.435767 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-kube-api-access-dh6td" pod "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" (UID: "2d8f6efb-ac1d-4d9a-9a19-f85f09e01760"). InnerVolumeSpecName "kube-api-access-dh6td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:18.533956 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.533924 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:18.533956 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.533952 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:18.533956 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.533961 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:18.534190 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.533969 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:18.534190 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.533979 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh6td\" (UniqueName: \"kubernetes.io/projected/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760-kube-api-access-dh6td\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:18.733047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.732957 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz"] Oct 30 19:49:18.744872 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:18.744827 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f6cef-openshift-default-78555bd8f7-xkknz"] Oct 30 19:49:19.051725 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:19.051639 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" path="/var/lib/kubelet/pods/2d8f6efb-ac1d-4d9a-9a19-f85f09e01760/volumes" Oct 30 19:49:20.743298 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.743260 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc"] Oct 30 19:49:20.744034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.743988 5047 cpu_manager.go:414] "containerMap: removing container" podUID="2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" containerName="istio-proxy" Oct 30 19:49:20.744034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.744018 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" containerName="istio-proxy" Oct 30 19:49:20.744034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.744034 5047 cpu_manager.go:414] "containerMap: removing container" podUID="2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" containerName="istio-proxy" Oct 30 19:49:20.744179 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.744039 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" containerName="istio-proxy" Oct 30 19:49:20.744179 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.744050 5047 cpu_manager.go:414] "containerMap: removing container" podUID="299d2abb-3201-42ac-8f33-edc45040171a" containerName="istio-proxy" Oct 30 19:49:20.744179 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.744055 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299d2abb-3201-42ac-8f33-edc45040171a" containerName="istio-proxy" Oct 30 19:49:20.744179 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.744088 5047 cpu_manager.go:414] "containerMap: removing container" podUID="299d2abb-3201-42ac-8f33-edc45040171a" containerName="istio-proxy" Oct 30 19:49:20.744179 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.744093 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299d2abb-3201-42ac-8f33-edc45040171a" containerName="istio-proxy" Oct 30 19:49:20.744352 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.744223 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" containerName="istio-proxy" Oct 30 19:49:20.744352 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.744230 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8f6efb-ac1d-4d9a-9a19-f85f09e01760" containerName="istio-proxy" Oct 30 19:49:20.744352 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.744236 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="299d2abb-3201-42ac-8f33-edc45040171a" containerName="istio-proxy" Oct 30 19:49:20.744352 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.744249 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="299d2abb-3201-42ac-8f33-edc45040171a" containerName="istio-proxy" Oct 30 19:49:20.749675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.749658 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" Oct 30 19:49:20.766910 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.766880 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc"] Oct 30 19:49:20.774049 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.771842 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-5250ee-dockercfg-tpvfb\"" Oct 30 19:49:20.774049 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.772445 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-5250ee\"" Oct 30 19:49:20.877027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.869369 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3475ab8d-e007-444d-baae-c24d13fd7a9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc\" (UID: \"3475ab8d-e007-444d-baae-c24d13fd7a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" Oct 30 19:49:20.877027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.869606 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9xr4\" (UniqueName: \"kubernetes.io/projected/3475ab8d-e007-444d-baae-c24d13fd7a9d-kube-api-access-t9xr4\") pod \"payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc\" (UID: \"3475ab8d-e007-444d-baae-c24d13fd7a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" Oct 30 19:49:20.973029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.972601 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3475ab8d-e007-444d-baae-c24d13fd7a9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc\" (UID: \"3475ab8d-e007-444d-baae-c24d13fd7a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" Oct 30 19:49:20.973029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.972863 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9xr4\" (UniqueName: \"kubernetes.io/projected/3475ab8d-e007-444d-baae-c24d13fd7a9d-kube-api-access-t9xr4\") pod \"payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc\" (UID: \"3475ab8d-e007-444d-baae-c24d13fd7a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" Oct 30 19:49:20.994761 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:20.987701 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3475ab8d-e007-444d-baae-c24d13fd7a9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc\" (UID: \"3475ab8d-e007-444d-baae-c24d13fd7a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" Oct 30 19:49:21.019071 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:21.015595 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d"] Oct 30 19:49:21.022259 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:21.022214 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9xr4\" (UniqueName: \"kubernetes.io/projected/3475ab8d-e007-444d-baae-c24d13fd7a9d-kube-api-access-t9xr4\") pod \"payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc\" (UID: \"3475ab8d-e007-444d-baae-c24d13fd7a9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" Oct 30 19:49:21.022498 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:21.022471 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d"] Oct 30 19:49:21.022652 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:21.022587 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" Oct 30 19:49:21.064581 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:21.064546 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" Oct 30 19:49:21.179054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:21.176113 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6zdb\" (UniqueName: \"kubernetes.io/projected/14cdb07d-1fb7-4959-baef-21ca0c5b51f1-kube-api-access-h6zdb\") pod \"payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d\" (UID: \"14cdb07d-1fb7-4959-baef-21ca0c5b51f1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" Oct 30 19:49:21.179054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:21.176260 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cdb07d-1fb7-4959-baef-21ca0c5b51f1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d\" (UID: \"14cdb07d-1fb7-4959-baef-21ca0c5b51f1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" Oct 30 19:49:21.279014 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:21.278917 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6zdb\" (UniqueName: \"kubernetes.io/projected/14cdb07d-1fb7-4959-baef-21ca0c5b51f1-kube-api-access-h6zdb\") pod \"payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d\" (UID: \"14cdb07d-1fb7-4959-baef-21ca0c5b51f1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" Oct 30 19:49:21.279170 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:21.279060 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cdb07d-1fb7-4959-baef-21ca0c5b51f1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d\" (UID: \"14cdb07d-1fb7-4959-baef-21ca0c5b51f1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" Oct 30 19:49:21.279773 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:21.279747 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cdb07d-1fb7-4959-baef-21ca0c5b51f1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d\" (UID: \"14cdb07d-1fb7-4959-baef-21ca0c5b51f1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" Oct 30 19:49:21.295603 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:21.295574 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6zdb\" (UniqueName: \"kubernetes.io/projected/14cdb07d-1fb7-4959-baef-21ca0c5b51f1-kube-api-access-h6zdb\") pod \"payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d\" (UID: \"14cdb07d-1fb7-4959-baef-21ca0c5b51f1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" Oct 30 19:49:21.358983 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:21.358953 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" Oct 30 19:49:21.364208 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:21.364178 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc"] Oct 30 19:49:21.364478 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:49:21.364452 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3475ab8d_e007_444d_baae_c24d13fd7a9d.slice/crio-2534674038d467157fec7ae71dc2ae23fd9fca509e923f064b9da641d0785481 WatchSource:0}: Error finding container 2534674038d467157fec7ae71dc2ae23fd9fca509e923f064b9da641d0785481: Status 404 returned error can't find the container with id 2534674038d467157fec7ae71dc2ae23fd9fca509e923f064b9da641d0785481 Oct 30 19:49:21.408544 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:21.408511 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" event={"ID":"3475ab8d-e007-444d-baae-c24d13fd7a9d","Type":"ContainerStarted","Data":"2534674038d467157fec7ae71dc2ae23fd9fca509e923f064b9da641d0785481"} Oct 30 19:49:21.664916 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:21.664883 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d"] Oct 30 19:49:21.665645 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:49:21.665606 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14cdb07d_1fb7_4959_baef_21ca0c5b51f1.slice/crio-b9bdda9317a763f044e918127059e83d328cde81f3e9a48505d65951655897ec WatchSource:0}: Error finding container b9bdda9317a763f044e918127059e83d328cde81f3e9a48505d65951655897ec: Status 404 returned error can't find the container with id b9bdda9317a763f044e918127059e83d328cde81f3e9a48505d65951655897ec Oct 30 19:49:22.417717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:22.417665 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" event={"ID":"14cdb07d-1fb7-4959-baef-21ca0c5b51f1","Type":"ContainerStarted","Data":"b5c30e03408f00e6322ef7454cc134976d667ebb4c330bc53073076ad4bdb91d"} Oct 30 19:49:22.418241 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:22.417740 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" Oct 30 19:49:22.418241 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:22.417768 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" event={"ID":"14cdb07d-1fb7-4959-baef-21ca0c5b51f1","Type":"ContainerStarted","Data":"b9bdda9317a763f044e918127059e83d328cde81f3e9a48505d65951655897ec"} Oct 30 19:49:22.419315 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:22.419290 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" event={"ID":"3475ab8d-e007-444d-baae-c24d13fd7a9d","Type":"ContainerStarted","Data":"7b3c74a68c63632d0dd97f1d32d4b2a8e10a66262f4244bd5971dd33db678edb"} Oct 30 19:49:22.419408 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:22.419341 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" Oct 30 19:49:22.455304 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:22.455257 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" podStartSLOduration=2.455244304 podStartE2EDuration="2.455244304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:22.452152863 +0000 UTC m=+2715.526608233" watchObservedRunningTime="2026-10-30 19:49:22.455244304 +0000 UTC m=+2715.529699661" Oct 30 19:49:22.489383 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:22.489338 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" podStartSLOduration=2.489327075 podStartE2EDuration="2.489327075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:22.485717548 +0000 UTC m=+2715.560172907" watchObservedRunningTime="2026-10-30 19:49:22.489327075 +0000 UTC m=+2715.563782432" Oct 30 19:49:22.757170 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:22.757085 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" podUID="bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:49:22.757324 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:22.757091 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" podUID="313dd358-14e7-4172-b9cb-3594dc0f4e7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:49:28.519997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.519595 5047 generic.go:356] "Generic (PLEG): container finished" podID="313dd358-14e7-4172-b9cb-3594dc0f4e7f" containerID="66c087809cb66c6882225218b8844fbb926c373583222388af34933bb7b2c072" exitCode=0 Oct 30 19:49:28.519997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.519665 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" event={"ID":"313dd358-14e7-4172-b9cb-3594dc0f4e7f","Type":"ContainerDied","Data":"66c087809cb66c6882225218b8844fbb926c373583222388af34933bb7b2c072"} Oct 30 19:49:28.522249 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.522083 5047 generic.go:356] "Generic (PLEG): container finished" podID="bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5" containerID="5d804544b322c0ed609cd511ad0520c8dbbec82473205a6fa60b81dfcfe63384" exitCode=0 Oct 30 19:49:28.522249 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.522166 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" event={"ID":"bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5","Type":"ContainerDied","Data":"5d804544b322c0ed609cd511ad0520c8dbbec82473205a6fa60b81dfcfe63384"} Oct 30 19:49:28.801859 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.801827 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" Oct 30 19:49:28.819724 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.819701 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" Oct 30 19:49:28.887035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.880645 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313dd358-14e7-4172-b9cb-3594dc0f4e7f-kube-api-access-smv5f\" (UniqueName: \"kubernetes.io/projected/313dd358-14e7-4172-b9cb-3594dc0f4e7f-kube-api-access-smv5f\") pod \"313dd358-14e7-4172-b9cb-3594dc0f4e7f\" (UID: \"313dd358-14e7-4172-b9cb-3594dc0f4e7f\") " Oct 30 19:49:28.887035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.880737 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/313dd358-14e7-4172-b9cb-3594dc0f4e7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313dd358-14e7-4172-b9cb-3594dc0f4e7f-plugins-config-volume\") pod \"313dd358-14e7-4172-b9cb-3594dc0f4e7f\" (UID: \"313dd358-14e7-4172-b9cb-3594dc0f4e7f\") " Oct 30 19:49:28.887035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.880832 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5-kube-api-access-cc8ss\" (UniqueName: \"kubernetes.io/projected/bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5-kube-api-access-cc8ss\") pod \"bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5\" (UID: \"bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5\") " Oct 30 19:49:28.887035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.880931 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5-plugins-config-volume\") pod \"bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5\" (UID: \"bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5\") " Oct 30 19:49:28.888029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.887681 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/313dd358-14e7-4172-b9cb-3594dc0f4e7f-plugins-config-volume" pod "313dd358-14e7-4172-b9cb-3594dc0f4e7f" (UID: "313dd358-14e7-4172-b9cb-3594dc0f4e7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:28.888674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.888629 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5-plugins-config-volume" pod "bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5" (UID: "bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:28.896541 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.896496 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5-kube-api-access-cc8ss" pod "bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5" (UID: "bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5"). InnerVolumeSpecName "kube-api-access-cc8ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:28.896668 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.896552 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313dd358-14e7-4172-b9cb-3594dc0f4e7f-kube-api-access-smv5f" pod "313dd358-14e7-4172-b9cb-3594dc0f4e7f" (UID: "313dd358-14e7-4172-b9cb-3594dc0f4e7f"). InnerVolumeSpecName "kube-api-access-smv5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:28.984089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.983336 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc8ss\" (UniqueName: \"kubernetes.io/projected/bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5-kube-api-access-cc8ss\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:28.984089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.983380 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:28.984089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.983397 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smv5f\" (UniqueName: \"kubernetes.io/projected/313dd358-14e7-4172-b9cb-3594dc0f4e7f-kube-api-access-smv5f\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:28.984089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:28.983414 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313dd358-14e7-4172-b9cb-3594dc0f4e7f-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:29.530547 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:29.530514 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" Oct 30 19:49:29.531056 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:29.530557 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr" event={"ID":"bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5","Type":"ContainerDied","Data":"b7112467f386e4eb6fe8d6b5ac921eaecace6b92416665cdec315cb88129cfef"} Oct 30 19:49:29.531056 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:29.530941 5047 scope.go:122] "RemoveContainer" containerID="5d804544b322c0ed609cd511ad0520c8dbbec82473205a6fa60b81dfcfe63384" Oct 30 19:49:29.532311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:29.532030 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" event={"ID":"313dd358-14e7-4172-b9cb-3594dc0f4e7f","Type":"ContainerDied","Data":"ebbda6a40eb6ffd273bf6e59531385e52c60305b6ef35a509b4ea880a8e09df0"} Oct 30 19:49:29.532311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:29.532072 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz" Oct 30 19:49:29.591321 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:29.591294 5047 scope.go:122] "RemoveContainer" containerID="66c087809cb66c6882225218b8844fbb926c373583222388af34933bb7b2c072" Oct 30 19:49:29.624358 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:29.624307 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr"] Oct 30 19:49:29.633291 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:29.633260 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f6cef-d7f56846b-j6bfr"] Oct 30 19:49:29.687952 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:29.687864 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz"] Oct 30 19:49:29.692455 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:29.692424 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f6cef-55bd7d749f-d8jzz"] Oct 30 19:49:31.054684 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:31.054647 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="313dd358-14e7-4172-b9cb-3594dc0f4e7f" path="/var/lib/kubelet/pods/313dd358-14e7-4172-b9cb-3594dc0f4e7f/volumes" Oct 30 19:49:31.055229 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:31.055212 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5" path="/var/lib/kubelet/pods/bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5/volumes" Oct 30 19:49:33.429723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:33.429685 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" Oct 30 19:49:33.431569 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:33.431546 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" Oct 30 19:49:33.996171 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:33.996128 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:49:33.996371 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:33.996209 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" podUID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:34.140495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:34.140453 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r"] Oct 30 19:49:34.604043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:34.603965 5047 generic.go:356] "Generic (PLEG): container finished" podID="ebafb097-ceaa-43ca-9ce4-023cdca5ad56" containerID="953e9794a673ece6ad8731c54373acb7f6362680a74e0be35a4a680dd8b1cdc7" exitCode=0 Oct 30 19:49:34.604043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:34.604027 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" event={"ID":"ebafb097-ceaa-43ca-9ce4-023cdca5ad56","Type":"ContainerDied","Data":"953e9794a673ece6ad8731c54373acb7f6362680a74e0be35a4a680dd8b1cdc7"} Oct 30 19:49:34.605103 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:34.604461 5047 scope.go:122] "RemoveContainer" containerID="953e9794a673ece6ad8731c54373acb7f6362680a74e0be35a4a680dd8b1cdc7" Oct 30 19:49:34.606512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:34.606480 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:34.606634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:34.606550 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:34.606634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:34.606580 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:35.612609 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:35.612579 5047 generic.go:356] "Generic (PLEG): container finished" podID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" containerID="37a8b3c2cdc3117e3568cf8c2b4d5041461096b417ae2d5356e70d2615c205c9" exitCode=0 Oct 30 19:49:35.613189 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:35.612648 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" event={"ID":"ad094cfc-29f5-44cc-82b8-162680c6d3a9","Type":"ContainerDied","Data":"37a8b3c2cdc3117e3568cf8c2b4d5041461096b417ae2d5356e70d2615c205c9"} Oct 30 19:49:35.613264 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:35.613237 5047 scope.go:122] "RemoveContainer" containerID="37a8b3c2cdc3117e3568cf8c2b4d5041461096b417ae2d5356e70d2615c205c9" Oct 30 19:49:35.614825 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:35.614804 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" event={"ID":"ebafb097-ceaa-43ca-9ce4-023cdca5ad56","Type":"ContainerStarted","Data":"de9c6eecd09c5a1a215b0b83256539dea6b4c3f883eca61bad9dd2c87acefabe"} Oct 30 19:49:35.615239 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:35.615211 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:35.615332 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:35.615276 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:35.615332 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:35.615307 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:36.452030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:36.447371 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:49:36.456028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:36.454063 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 19:49:36.456028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:36.454131 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" podUID="ebafb097-ceaa-43ca-9ce4-023cdca5ad56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 19:49:36.624647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:36.624612 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" event={"ID":"ad094cfc-29f5-44cc-82b8-162680c6d3a9","Type":"ContainerStarted","Data":"50221a7adee5a4075732d0f4760c83dc87c0f34951cdea7f3c1703fdde09067d"} Oct 30 19:49:36.865467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:36.865425 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:49:36.865863 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:36.865834 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 19:49:36.866060 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:36.866036 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" podUID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 19:49:37.455903 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:37.455425 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 19:49:37.455903 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:37.455494 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" podUID="ebafb097-ceaa-43ca-9ce4-023cdca5ad56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 19:49:37.865436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:37.865394 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 19:49:37.866079 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:37.866041 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" podUID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 19:49:38.450028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:38.449368 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 19:49:38.450028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:38.449434 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" podUID="ebafb097-ceaa-43ca-9ce4-023cdca5ad56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 19:49:38.869031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:38.865424 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 19:49:38.869031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:38.865489 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" podUID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 19:49:39.179164 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:39.178787 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" podUID="90483a7b-6fe0-499d-abbf-7e0461ebd934" containerName="payload-pre-processing" containerID="cri-o://3384b4ba5193e56db2f13522c01a6b68cc10c0f43254cc70c6a9e2a493dc9126" gracePeriod=25 Oct 30 19:49:39.444869 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:39.444371 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 19:49:39.444869 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:39.444437 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" podUID="ebafb097-ceaa-43ca-9ce4-023cdca5ad56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 19:49:39.612137 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:49:39.612102 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:49:39.659030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:39.658106 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:49:39.871030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:39.868536 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 19:49:39.871030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:39.868626 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" podUID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 19:49:40.125791 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:40.125697 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" podUID="90483a7b-6fe0-499d-abbf-7e0461ebd934" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:49:40.480088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:40.475142 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 30 19:49:40.480088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:40.475210 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" podUID="ebafb097-ceaa-43ca-9ce4-023cdca5ad56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 30 19:49:40.868622 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:40.868443 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 19:49:40.868622 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:40.868526 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" podUID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 19:49:41.109045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.102281 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d"] Oct 30 19:49:41.109045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.103466 5047 cpu_manager.go:414] "containerMap: removing container" podUID="bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5" containerName="payload-processing" Oct 30 19:49:41.109045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.103489 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5" containerName="payload-processing" Oct 30 19:49:41.109045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.103508 5047 cpu_manager.go:414] "containerMap: removing container" podUID="313dd358-14e7-4172-b9cb-3594dc0f4e7f" containerName="payload-pre-processing" Oct 30 19:49:41.109045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.103516 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313dd358-14e7-4172-b9cb-3594dc0f4e7f" containerName="payload-pre-processing" Oct 30 19:49:41.109045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.103718 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="bedb0dd5-4c08-4e9f-ae37-17ee9a39a2f5" containerName="payload-processing" Oct 30 19:49:41.109045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.103736 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="313dd358-14e7-4172-b9cb-3594dc0f4e7f" containerName="payload-pre-processing" Oct 30 19:49:41.122489 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.122377 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" Oct 30 19:49:41.168535 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.168495 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-54086b-dockercfg-nldlt\"" Oct 30 19:49:41.168750 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.168639 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-54086b\"" Oct 30 19:49:41.175933 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.175896 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d"] Oct 30 19:49:41.213071 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.213038 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7c056d-fb1d-49f8-9371-be32ecfdf069-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d\" (UID: \"ed7c056d-fb1d-49f8-9371-be32ecfdf069\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" Oct 30 19:49:41.213071 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.213075 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zczt7\" (UniqueName: \"kubernetes.io/projected/ed7c056d-fb1d-49f8-9371-be32ecfdf069-kube-api-access-zczt7\") pod \"payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d\" (UID: \"ed7c056d-fb1d-49f8-9371-be32ecfdf069\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" Oct 30 19:49:41.314817 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.314779 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7c056d-fb1d-49f8-9371-be32ecfdf069-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d\" (UID: \"ed7c056d-fb1d-49f8-9371-be32ecfdf069\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" Oct 30 19:49:41.314990 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.314829 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zczt7\" (UniqueName: \"kubernetes.io/projected/ed7c056d-fb1d-49f8-9371-be32ecfdf069-kube-api-access-zczt7\") pod \"payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d\" (UID: \"ed7c056d-fb1d-49f8-9371-be32ecfdf069\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" Oct 30 19:49:41.315784 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.315751 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7c056d-fb1d-49f8-9371-be32ecfdf069-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d\" (UID: \"ed7c056d-fb1d-49f8-9371-be32ecfdf069\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" Oct 30 19:49:41.357026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.356512 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zczt7\" (UniqueName: \"kubernetes.io/projected/ed7c056d-fb1d-49f8-9371-be32ecfdf069-kube-api-access-zczt7\") pod \"payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d\" (UID: \"ed7c056d-fb1d-49f8-9371-be32ecfdf069\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" Oct 30 19:49:41.464088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.461377 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:49:41.464560 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.464523 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 19:49:41.484675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.484651 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" Oct 30 19:49:41.865711 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.865195 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 19:49:41.865711 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.865255 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" podUID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 19:49:41.868408 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.868103 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d"] Oct 30 19:49:41.874059 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:41.874040 5047 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:49:42.774797 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:42.774764 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" event={"ID":"ed7c056d-fb1d-49f8-9371-be32ecfdf069","Type":"ContainerStarted","Data":"b71b792c0946ab3798e6dc968237c8b669065d83617a62b1cd5e8f032de921b3"} Oct 30 19:49:42.869915 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:42.869880 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:49:42.870421 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:42.870394 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:49:42.871036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:42.870995 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:49:43.783995 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:43.783960 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" event={"ID":"ed7c056d-fb1d-49f8-9371-be32ecfdf069","Type":"ContainerStarted","Data":"78fac3604b7086db0e0ffb53802d88fd974f76bc102dace7b5147fd569fee207"} Oct 30 19:49:43.784381 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:43.784081 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" Oct 30 19:49:43.820600 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:43.820537 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" podStartSLOduration=2.12072143 podStartE2EDuration="3.820519536s" totalImagesPullingTime="1.699798106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:43.8172999 +0000 UTC m=+2736.891755259" watchObservedRunningTime="2026-10-30 19:49:43.820519536 +0000 UTC m=+2736.894974899" Oct 30 19:49:48.765762 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.765665 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz"] Oct 30 19:49:48.770561 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.770537 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.788821 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.788439 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-5a0a17-openshift-default-dockercfg-xfwqt\"" Oct 30 19:49:48.816882 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.816845 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz"] Oct 30 19:49:48.844993 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.844957 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pznvl\" (UniqueName: \"kubernetes.io/projected/f3712046-b84a-45fd-b20f-49c8d68adc11-kube-api-access-pznvl\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.845202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.845030 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3712046-b84a-45fd-b20f-49c8d68adc11-istiod-ca-cert\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.845202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.845156 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-workload-certs\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.845325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.845205 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-token\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.845325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.845238 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-workload-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.845430 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.845383 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-envoy\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.845487 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.845449 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-podinfo\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.845571 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.845555 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-data\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.845629 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.845584 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-credential-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.956520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.954114 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-envoy\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.956520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.954221 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-podinfo\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.956520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.954343 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-data\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.956520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.954376 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-credential-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.956520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.954490 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pznvl\" (UniqueName: \"kubernetes.io/projected/f3712046-b84a-45fd-b20f-49c8d68adc11-kube-api-access-pznvl\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.956520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.954540 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3712046-b84a-45fd-b20f-49c8d68adc11-istiod-ca-cert\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.956520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.954592 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-workload-certs\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.956520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.954628 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-token\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.956520 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.954708 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-workload-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.979026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.969650 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-workload-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.979026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.970400 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-credential-socket\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.979026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.970412 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3712046-b84a-45fd-b20f-49c8d68adc11-istiod-ca-cert\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.979026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.970734 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-data\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.994024 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.990174 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-workload-certs\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.994024 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.992052 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-token\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.994024 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.992625 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-podinfo\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.994905 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.994876 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-envoy\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:48.998027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:48.996270 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pznvl\" (UniqueName: \"kubernetes.io/projected/f3712046-b84a-45fd-b20f-49c8d68adc11-kube-api-access-pznvl\") pod \"e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:49.114241 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:49.114207 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:49.474526 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:49:49.474498 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3712046_b84a_45fd_b20f_49c8d68adc11.slice/crio-0ec91d0363852b2f359dc12a0813ee3d24a6231a24a2fc567fe3afdcadd5bbd2 WatchSource:0}: Error finding container 0ec91d0363852b2f359dc12a0813ee3d24a6231a24a2fc567fe3afdcadd5bbd2: Status 404 returned error can't find the container with id 0ec91d0363852b2f359dc12a0813ee3d24a6231a24a2fc567fe3afdcadd5bbd2 Oct 30 19:49:49.474644 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:49.474599 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz"] Oct 30 19:49:49.477996 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:49.477959 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:49.478095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:49.478065 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:49.478135 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:49.478109 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:49.830770 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:49.830736 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" event={"ID":"f3712046-b84a-45fd-b20f-49c8d68adc11","Type":"ContainerStarted","Data":"64ce6678e99049f8c234669acbc47faf6ef5c2089a4f8c4b5806331dd25ede7b"} Oct 30 19:49:49.831257 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:49.830792 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" event={"ID":"f3712046-b84a-45fd-b20f-49c8d68adc11","Type":"ContainerStarted","Data":"0ec91d0363852b2f359dc12a0813ee3d24a6231a24a2fc567fe3afdcadd5bbd2"} Oct 30 19:49:49.871513 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:49.871459 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" podStartSLOduration=1.871444775 podStartE2EDuration="1.871444775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:49.867266004 +0000 UTC m=+2742.941721362" watchObservedRunningTime="2026-10-30 19:49:49.871444775 +0000 UTC m=+2742.945900211" Oct 30 19:49:50.114464 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:50.114385 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:50.121728 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:50.121701 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:50.125639 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:50.125600 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" podUID="90483a7b-6fe0-499d-abbf-7e0461ebd934" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:49:50.837915 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:50.837880 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:50.839094 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:50.839072 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:49:54.793177 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:54.793139 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" Oct 30 19:49:58.921869 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:58.921835 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt"] Oct 30 19:49:58.928708 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:58.928686 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:58.949428 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:58.941816 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-852709-openshift-default-dockercfg-ps9g6\"" Oct 30 19:49:58.958222 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:58.958058 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt"] Oct 30 19:49:59.102568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.102521 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-credential-socket\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.102865 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.102832 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-podinfo\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.103045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.103028 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-workload-certs\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.103190 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.103173 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-data\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.103334 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.103317 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6lm5\" (UniqueName: \"kubernetes.io/projected/f5b67c1b-d500-4e75-affb-e748e37b0cfb-kube-api-access-b6lm5\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.103468 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.103454 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istiod-ca-cert\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.103596 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.103582 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-envoy\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.103871 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.103854 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-token\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.104027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.103990 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-workload-socket\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.214076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.210981 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-workload-socket\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.214076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.211280 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-credential-socket\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.214076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.211314 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-podinfo\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.214076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.211371 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-workload-certs\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.214076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.211418 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-data\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.214076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.211465 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6lm5\" (UniqueName: \"kubernetes.io/projected/f5b67c1b-d500-4e75-affb-e748e37b0cfb-kube-api-access-b6lm5\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.214076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.211526 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istiod-ca-cert\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.214076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.211577 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-envoy\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.214076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.211781 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-token\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.214076 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.213542 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-workload-certs\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.220134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.214901 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-credential-socket\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.220134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.215757 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istiod-ca-cert\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.220134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.216411 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-data\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.220134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.216346 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-workload-socket\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.222077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.222032 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-podinfo\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.239573 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.237437 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-envoy\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.270813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.270772 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6lm5\" (UniqueName: \"kubernetes.io/projected/f5b67c1b-d500-4e75-affb-e748e37b0cfb-kube-api-access-b6lm5\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.272515 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.272489 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-token\") pod \"e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.549656 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.549624 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" Oct 30 19:49:59.552759 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.552725 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:49:59.642020 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.641973 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90483a7b-6fe0-499d-abbf-7e0461ebd934-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90483a7b-6fe0-499d-abbf-7e0461ebd934-plugins-config-volume\") pod \"90483a7b-6fe0-499d-abbf-7e0461ebd934\" (UID: \"90483a7b-6fe0-499d-abbf-7e0461ebd934\") " Oct 30 19:49:59.642191 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.642030 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90483a7b-6fe0-499d-abbf-7e0461ebd934-kube-api-access-2q54f\" (UniqueName: \"kubernetes.io/projected/90483a7b-6fe0-499d-abbf-7e0461ebd934-kube-api-access-2q54f\") pod \"90483a7b-6fe0-499d-abbf-7e0461ebd934\" (UID: \"90483a7b-6fe0-499d-abbf-7e0461ebd934\") " Oct 30 19:49:59.643112 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.643087 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90483a7b-6fe0-499d-abbf-7e0461ebd934-plugins-config-volume" pod "90483a7b-6fe0-499d-abbf-7e0461ebd934" (UID: "90483a7b-6fe0-499d-abbf-7e0461ebd934"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:59.645712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.645682 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90483a7b-6fe0-499d-abbf-7e0461ebd934-kube-api-access-2q54f" pod "90483a7b-6fe0-499d-abbf-7e0461ebd934" (UID: "90483a7b-6fe0-499d-abbf-7e0461ebd934"). InnerVolumeSpecName "kube-api-access-2q54f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:59.743993 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.743908 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90483a7b-6fe0-499d-abbf-7e0461ebd934-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:59.743993 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.743946 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q54f\" (UniqueName: \"kubernetes.io/projected/90483a7b-6fe0-499d-abbf-7e0461ebd934-kube-api-access-2q54f\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:49:59.871189 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:49:59.871160 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b67c1b_d500_4e75_affb_e748e37b0cfb.slice/crio-b232a20495b70c172db8c96e3cc0fbf51c8fc2da3c5b0ec80c58c079c99ddb9e WatchSource:0}: Error finding container b232a20495b70c172db8c96e3cc0fbf51c8fc2da3c5b0ec80c58c079c99ddb9e: Status 404 returned error can't find the container with id b232a20495b70c172db8c96e3cc0fbf51c8fc2da3c5b0ec80c58c079c99ddb9e Oct 30 19:49:59.871377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.871352 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt"] Oct 30 19:49:59.874188 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.874157 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:59.874280 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.874220 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:59.874280 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.874249 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:59.913199 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.913169 5047 generic.go:356] "Generic (PLEG): container finished" podID="90483a7b-6fe0-499d-abbf-7e0461ebd934" containerID="3384b4ba5193e56db2f13522c01a6b68cc10c0f43254cc70c6a9e2a493dc9126" exitCode=0 Oct 30 19:49:59.913354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.913238 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" Oct 30 19:49:59.913354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.913262 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" event={"ID":"90483a7b-6fe0-499d-abbf-7e0461ebd934","Type":"ContainerDied","Data":"3384b4ba5193e56db2f13522c01a6b68cc10c0f43254cc70c6a9e2a493dc9126"} Oct 30 19:49:59.913354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.913321 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r" event={"ID":"90483a7b-6fe0-499d-abbf-7e0461ebd934","Type":"ContainerDied","Data":"a1b0d159b93bc6e36aca75656941ca11e1d466473e23c338a7451d7520d69fcd"} Oct 30 19:49:59.913354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.913347 5047 scope.go:122] "RemoveContainer" containerID="3384b4ba5193e56db2f13522c01a6b68cc10c0f43254cc70c6a9e2a493dc9126" Oct 30 19:49:59.917475 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.917444 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" event={"ID":"f5b67c1b-d500-4e75-affb-e748e37b0cfb","Type":"ContainerStarted","Data":"b232a20495b70c172db8c96e3cc0fbf51c8fc2da3c5b0ec80c58c079c99ddb9e"} Oct 30 19:49:59.925191 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.925174 5047 scope.go:122] "RemoveContainer" containerID="3384b4ba5193e56db2f13522c01a6b68cc10c0f43254cc70c6a9e2a493dc9126" Oct 30 19:49:59.925598 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:49:59.925438 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3384b4ba5193e56db2f13522c01a6b68cc10c0f43254cc70c6a9e2a493dc9126\": container with ID starting with 3384b4ba5193e56db2f13522c01a6b68cc10c0f43254cc70c6a9e2a493dc9126 not found: ID does not exist" containerID="3384b4ba5193e56db2f13522c01a6b68cc10c0f43254cc70c6a9e2a493dc9126" Oct 30 19:49:59.925598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.925466 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3384b4ba5193e56db2f13522c01a6b68cc10c0f43254cc70c6a9e2a493dc9126"} err="failed to get container status \"3384b4ba5193e56db2f13522c01a6b68cc10c0f43254cc70c6a9e2a493dc9126\": rpc error: code = NotFound desc = could not find container \"3384b4ba5193e56db2f13522c01a6b68cc10c0f43254cc70c6a9e2a493dc9126\": container with ID starting with 3384b4ba5193e56db2f13522c01a6b68cc10c0f43254cc70c6a9e2a493dc9126 not found: ID does not exist" Oct 30 19:49:59.966020 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.965981 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r"] Oct 30 19:49:59.969602 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:49:59.969578 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-576a62-5bb8c998f9-c5b4r"] Oct 30 19:50:00.929040 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:00.928980 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" event={"ID":"f5b67c1b-d500-4e75-affb-e748e37b0cfb","Type":"ContainerStarted","Data":"d3033c99f6d6304cf6d9b723014b787b4375f0c84c2a148a5945cee80f16fe82"} Oct 30 19:50:01.052620 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:01.052586 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90483a7b-6fe0-499d-abbf-7e0461ebd934" path="/var/lib/kubelet/pods/90483a7b-6fe0-499d-abbf-7e0461ebd934/volumes" Oct 30 19:50:01.553025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:01.552976 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:50:01.559525 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:01.559492 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:50:01.613445 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:01.613370 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" podStartSLOduration=3.613346788 podStartE2EDuration="3.613346788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:00.984841295 +0000 UTC m=+2754.059296652" watchObservedRunningTime="2026-10-30 19:50:01.613346788 +0000 UTC m=+2754.687802147" Oct 30 19:50:01.936618 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:01.936582 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:50:01.937647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:01.937624 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:50:06.323593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:06.323547 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft"] Oct 30 19:50:06.324674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:06.324638 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" podUID="980112bf-3559-4a54-aea8-4ded5e0f269e" containerName="maas-api" containerID="cri-o://27d511ae0bcd590d90fb59331c6d8433029f06bfd8834dc50ec23fc00cf5c7ef" gracePeriod=30 Oct 30 19:50:06.995030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:06.992103 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:50:07.016955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.016923 5047 generic.go:356] "Generic (PLEG): container finished" podID="980112bf-3559-4a54-aea8-4ded5e0f269e" containerID="27d511ae0bcd590d90fb59331c6d8433029f06bfd8834dc50ec23fc00cf5c7ef" exitCode=0 Oct 30 19:50:07.017168 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.016982 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" Oct 30 19:50:07.017168 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.017046 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" event={"ID":"980112bf-3559-4a54-aea8-4ded5e0f269e","Type":"ContainerDied","Data":"27d511ae0bcd590d90fb59331c6d8433029f06bfd8834dc50ec23fc00cf5c7ef"} Oct 30 19:50:07.017168 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.017116 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft" event={"ID":"980112bf-3559-4a54-aea8-4ded5e0f269e","Type":"ContainerDied","Data":"e8c46c9c15b4cf6e4836e4c1e47e266c7f8dfe4984f07d6dbeada5fc15bb13c5"} Oct 30 19:50:07.017168 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.017148 5047 scope.go:122] "RemoveContainer" containerID="27d511ae0bcd590d90fb59331c6d8433029f06bfd8834dc50ec23fc00cf5c7ef" Oct 30 19:50:07.033968 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.033938 5047 scope.go:122] "RemoveContainer" containerID="27d511ae0bcd590d90fb59331c6d8433029f06bfd8834dc50ec23fc00cf5c7ef" Oct 30 19:50:07.035505 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:50:07.035470 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d511ae0bcd590d90fb59331c6d8433029f06bfd8834dc50ec23fc00cf5c7ef\": container with ID starting with 27d511ae0bcd590d90fb59331c6d8433029f06bfd8834dc50ec23fc00cf5c7ef not found: ID does not exist" containerID="27d511ae0bcd590d90fb59331c6d8433029f06bfd8834dc50ec23fc00cf5c7ef" Oct 30 19:50:07.035643 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.035515 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d511ae0bcd590d90fb59331c6d8433029f06bfd8834dc50ec23fc00cf5c7ef"} err="failed to get container status \"27d511ae0bcd590d90fb59331c6d8433029f06bfd8834dc50ec23fc00cf5c7ef\": rpc error: code = NotFound desc = could not find container \"27d511ae0bcd590d90fb59331c6d8433029f06bfd8834dc50ec23fc00cf5c7ef\": container with ID starting with 27d511ae0bcd590d90fb59331c6d8433029f06bfd8834dc50ec23fc00cf5c7ef not found: ID does not exist" Oct 30 19:50:07.148847 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.148763 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/980112bf-3559-4a54-aea8-4ded5e0f269e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/980112bf-3559-4a54-aea8-4ded5e0f269e-metrics-certs\") pod \"980112bf-3559-4a54-aea8-4ded5e0f269e\" (UID: \"980112bf-3559-4a54-aea8-4ded5e0f269e\") " Oct 30 19:50:07.149023 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.148858 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/980112bf-3559-4a54-aea8-4ded5e0f269e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/980112bf-3559-4a54-aea8-4ded5e0f269e-openshift-service-ca\") pod \"980112bf-3559-4a54-aea8-4ded5e0f269e\" (UID: \"980112bf-3559-4a54-aea8-4ded5e0f269e\") " Oct 30 19:50:07.149023 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.148906 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/980112bf-3559-4a54-aea8-4ded5e0f269e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/980112bf-3559-4a54-aea8-4ded5e0f269e-maas-api-tls\") pod \"980112bf-3559-4a54-aea8-4ded5e0f269e\" (UID: \"980112bf-3559-4a54-aea8-4ded5e0f269e\") " Oct 30 19:50:07.149023 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.148968 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/980112bf-3559-4a54-aea8-4ded5e0f269e-kube-api-access-xm9n8\" (UniqueName: \"kubernetes.io/projected/980112bf-3559-4a54-aea8-4ded5e0f269e-kube-api-access-xm9n8\") pod \"980112bf-3559-4a54-aea8-4ded5e0f269e\" (UID: \"980112bf-3559-4a54-aea8-4ded5e0f269e\") " Oct 30 19:50:07.149279 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.149249 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980112bf-3559-4a54-aea8-4ded5e0f269e-openshift-service-ca" pod "980112bf-3559-4a54-aea8-4ded5e0f269e" (UID: "980112bf-3559-4a54-aea8-4ded5e0f269e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:07.149955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.149928 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/980112bf-3559-4a54-aea8-4ded5e0f269e-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:50:07.151239 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.151212 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/980112bf-3559-4a54-aea8-4ded5e0f269e-metrics-certs" pod "980112bf-3559-4a54-aea8-4ded5e0f269e" (UID: "980112bf-3559-4a54-aea8-4ded5e0f269e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:07.151391 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.151367 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/980112bf-3559-4a54-aea8-4ded5e0f269e-maas-api-tls" pod "980112bf-3559-4a54-aea8-4ded5e0f269e" (UID: "980112bf-3559-4a54-aea8-4ded5e0f269e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:07.151545 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.151525 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980112bf-3559-4a54-aea8-4ded5e0f269e-kube-api-access-xm9n8" pod "980112bf-3559-4a54-aea8-4ded5e0f269e" (UID: "980112bf-3559-4a54-aea8-4ded5e0f269e"). InnerVolumeSpecName "kube-api-access-xm9n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:07.251593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.251556 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980112bf-3559-4a54-aea8-4ded5e0f269e-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:50:07.251593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.251586 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/980112bf-3559-4a54-aea8-4ded5e0f269e-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:50:07.251593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.251596 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm9n8\" (UniqueName: \"kubernetes.io/projected/980112bf-3559-4a54-aea8-4ded5e0f269e-kube-api-access-xm9n8\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:50:07.388889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.388851 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft"] Oct 30 19:50:07.393193 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:07.393163 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-576a62-55f5785447-j2vft"] Oct 30 19:50:09.052272 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:09.052240 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="980112bf-3559-4a54-aea8-4ded5e0f269e" path="/var/lib/kubelet/pods/980112bf-3559-4a54-aea8-4ded5e0f269e/volumes" Oct 30 19:50:09.602599 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:50:09.602553 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:50:14.254867 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:14.254825 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd"] Oct 30 19:50:14.255401 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:14.255074 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" podUID="df7012d4-5d6b-4586-a4ac-629154c1d347" containerName="istio-proxy" containerID="cri-o://7663aa77ca4c9b8e5823ae42f1ecb32daaf4303d1bde9a086a06e1b92f97baf1" gracePeriod=30 Oct 30 19:50:17.210518 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.210482 5047 patch_prober.go:29] interesting pod/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:50:17.210892 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.210553 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" podUID="df7012d4-5d6b-4586-a4ac-629154c1d347" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:50:17.683850 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.683811 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q"] Oct 30 19:50:17.684647 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.684629 5047 cpu_manager.go:414] "containerMap: removing container" podUID="980112bf-3559-4a54-aea8-4ded5e0f269e" containerName="maas-api" Oct 30 19:50:17.684703 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.684648 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="980112bf-3559-4a54-aea8-4ded5e0f269e" containerName="maas-api" Oct 30 19:50:17.684703 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.684695 5047 cpu_manager.go:414] "containerMap: removing container" podUID="90483a7b-6fe0-499d-abbf-7e0461ebd934" containerName="payload-pre-processing" Oct 30 19:50:17.684703 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.684701 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90483a7b-6fe0-499d-abbf-7e0461ebd934" containerName="payload-pre-processing" Oct 30 19:50:17.684817 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.684807 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="90483a7b-6fe0-499d-abbf-7e0461ebd934" containerName="payload-pre-processing" Oct 30 19:50:17.684862 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.684824 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="980112bf-3559-4a54-aea8-4ded5e0f269e" containerName="maas-api" Oct 30 19:50:17.690141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.690113 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.712054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.712000 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q"] Oct 30 19:50:17.715846 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.715817 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-cd4bde\"/\"default-dockercfg-tns5f\"" Oct 30 19:50:17.716000 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.715856 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-cd4bde\"/\"openshift-service-ca.crt\"" Oct 30 19:50:17.719501 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.719476 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-cd4bde\"/\"kube-root-ca.crt\"" Oct 30 19:50:17.719737 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.719693 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-cd4bde\"/\"auth-test-model-cd4bde-kserve-self-signed-certs\"" Oct 30 19:50:17.796994 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.796956 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-home\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.797196 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.797026 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-dshm\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.797297 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.797259 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-tmp-dir\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.797402 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.797377 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847ac050-31eb-4adb-9457-2daf3e4ae09b-tls-certs\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.797685 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.797670 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-model-cache\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.797750 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.797734 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqcb2\" (UniqueName: \"kubernetes.io/projected/847ac050-31eb-4adb-9457-2daf3e4ae09b-kube-api-access-gqcb2\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.899070 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.899033 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-model-cache\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.899245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.899108 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqcb2\" (UniqueName: \"kubernetes.io/projected/847ac050-31eb-4adb-9457-2daf3e4ae09b-kube-api-access-gqcb2\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.899245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.899156 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-home\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.899245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.899187 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-dshm\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.899398 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.899255 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-tmp-dir\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.899398 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.899284 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847ac050-31eb-4adb-9457-2daf3e4ae09b-tls-certs\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.899482 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.899453 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-model-cache\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.899617 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.899596 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-home\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.899685 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.899653 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-tmp-dir\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.901430 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.901411 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-dshm\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.901819 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.901797 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847ac050-31eb-4adb-9457-2daf3e4ae09b-tls-certs\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:17.916231 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:17.916202 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqcb2\" (UniqueName: \"kubernetes.io/projected/847ac050-31eb-4adb-9457-2daf3e4ae09b-kube-api-access-gqcb2\") pod \"auth-test-model-cd4bde-kserve-5bc7456557-9qs4q\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:18.075215 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:18.075105 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:18.411743 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:18.411710 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q"] Oct 30 19:50:18.412096 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:50:18.411867 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod847ac050_31eb_4adb_9457_2daf3e4ae09b.slice/crio-3b667a59e667547a43be9411719b7aa3ce003656ab8c1b2f6b32c80f76ff8968 WatchSource:0}: Error finding container 3b667a59e667547a43be9411719b7aa3ce003656ab8c1b2f6b32c80f76ff8968: Status 404 returned error can't find the container with id 3b667a59e667547a43be9411719b7aa3ce003656ab8c1b2f6b32c80f76ff8968 Oct 30 19:50:19.246190 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.246158 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" event={"ID":"847ac050-31eb-4adb-9457-2daf3e4ae09b","Type":"ContainerStarted","Data":"4d51faca20b3e3bd6a5333b80ed6653439c5145ac9b10be150e3c771d18514e9"} Oct 30 19:50:19.246367 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.246200 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" event={"ID":"847ac050-31eb-4adb-9457-2daf3e4ae09b","Type":"ContainerStarted","Data":"3b667a59e667547a43be9411719b7aa3ce003656ab8c1b2f6b32c80f76ff8968"} Oct 30 19:50:19.282311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.282259 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" podStartSLOduration=2.282244376 podStartE2EDuration="2.282244376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:19.28077835 +0000 UTC m=+2772.355233708" watchObservedRunningTime="2026-10-30 19:50:19.282244376 +0000 UTC m=+2772.356699732" Oct 30 19:50:19.590643 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.590612 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:50:19.647403 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.647366 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-token\" (UniqueName: \"kubernetes.io/projected/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-token\") pod \"df7012d4-5d6b-4586-a4ac-629154c1d347\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " Oct 30 19:50:19.647588 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.647412 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-workload-certs\") pod \"df7012d4-5d6b-4586-a4ac-629154c1d347\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " Oct 30 19:50:19.647588 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.647462 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7012d4-5d6b-4586-a4ac-629154c1d347-kube-api-access-lfbzt\" (UniqueName: \"kubernetes.io/projected/df7012d4-5d6b-4586-a4ac-629154c1d347-kube-api-access-lfbzt\") pod \"df7012d4-5d6b-4586-a4ac-629154c1d347\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " Oct 30 19:50:19.647588 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.647504 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-data\") pod \"df7012d4-5d6b-4586-a4ac-629154c1d347\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " Oct 30 19:50:19.647588 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.647568 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-envoy\") pod \"df7012d4-5d6b-4586-a4ac-629154c1d347\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " Oct 30 19:50:19.647588 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.647589 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-workload-socket\") pod \"df7012d4-5d6b-4586-a4ac-629154c1d347\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " Oct 30 19:50:19.647876 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.647621 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-credential-socket\") pod \"df7012d4-5d6b-4586-a4ac-629154c1d347\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " Oct 30 19:50:19.647876 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.647743 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-podinfo\") pod \"df7012d4-5d6b-4586-a4ac-629154c1d347\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " Oct 30 19:50:19.647876 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.647779 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-workload-certs" pod "df7012d4-5d6b-4586-a4ac-629154c1d347" (UID: "df7012d4-5d6b-4586-a4ac-629154c1d347"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:19.647876 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.647807 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df7012d4-5d6b-4586-a4ac-629154c1d347-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7012d4-5d6b-4586-a4ac-629154c1d347-istiod-ca-cert\") pod \"df7012d4-5d6b-4586-a4ac-629154c1d347\" (UID: \"df7012d4-5d6b-4586-a4ac-629154c1d347\") " Oct 30 19:50:19.648108 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.648050 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-credential-socket" pod "df7012d4-5d6b-4586-a4ac-629154c1d347" (UID: "df7012d4-5d6b-4586-a4ac-629154c1d347"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:19.648264 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.648245 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-workload-socket" pod "df7012d4-5d6b-4586-a4ac-629154c1d347" (UID: "df7012d4-5d6b-4586-a4ac-629154c1d347"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:19.648985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.648686 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:50:19.648985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.648714 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:50:19.648985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.648727 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:50:19.648985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.648760 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-data" pod "df7012d4-5d6b-4586-a4ac-629154c1d347" (UID: "df7012d4-5d6b-4586-a4ac-629154c1d347"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:19.649298 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.649248 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df7012d4-5d6b-4586-a4ac-629154c1d347-istiod-ca-cert" pod "df7012d4-5d6b-4586-a4ac-629154c1d347" (UID: "df7012d4-5d6b-4586-a4ac-629154c1d347"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:19.650121 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.650090 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-envoy" pod "df7012d4-5d6b-4586-a4ac-629154c1d347" (UID: "df7012d4-5d6b-4586-a4ac-629154c1d347"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:19.650217 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.650185 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7012d4-5d6b-4586-a4ac-629154c1d347-kube-api-access-lfbzt" pod "df7012d4-5d6b-4586-a4ac-629154c1d347" (UID: "df7012d4-5d6b-4586-a4ac-629154c1d347"). InnerVolumeSpecName "kube-api-access-lfbzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:19.650268 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.650218 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-token" pod "df7012d4-5d6b-4586-a4ac-629154c1d347" (UID: "df7012d4-5d6b-4586-a4ac-629154c1d347"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:19.650805 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.650780 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-podinfo" pod "df7012d4-5d6b-4586-a4ac-629154c1d347" (UID: "df7012d4-5d6b-4586-a4ac-629154c1d347"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:50:19.750670 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.750638 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7012d4-5d6b-4586-a4ac-629154c1d347-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:50:19.750670 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.750668 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:50:19.750670 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.750677 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfbzt\" (UniqueName: \"kubernetes.io/projected/df7012d4-5d6b-4586-a4ac-629154c1d347-kube-api-access-lfbzt\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:50:19.750902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.750687 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:50:19.750902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.750696 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:50:19.750902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:19.750703 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7012d4-5d6b-4586-a4ac-629154c1d347-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:50:20.076234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:20.076201 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:20.255502 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:20.255466 5047 generic.go:356] "Generic (PLEG): container finished" podID="df7012d4-5d6b-4586-a4ac-629154c1d347" containerID="7663aa77ca4c9b8e5823ae42f1ecb32daaf4303d1bde9a086a06e1b92f97baf1" exitCode=0 Oct 30 19:50:20.255689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:20.255522 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" Oct 30 19:50:20.255689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:20.255548 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" event={"ID":"df7012d4-5d6b-4586-a4ac-629154c1d347","Type":"ContainerDied","Data":"7663aa77ca4c9b8e5823ae42f1ecb32daaf4303d1bde9a086a06e1b92f97baf1"} Oct 30 19:50:20.255689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:20.255600 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd" event={"ID":"df7012d4-5d6b-4586-a4ac-629154c1d347","Type":"ContainerDied","Data":"baad2ecbe9fdec62ad904c7e4cc8b305482dbe25937d9f42ce81e8b7540c48f3"} Oct 30 19:50:20.255689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:20.255623 5047 scope.go:122] "RemoveContainer" containerID="7663aa77ca4c9b8e5823ae42f1ecb32daaf4303d1bde9a086a06e1b92f97baf1" Oct 30 19:50:20.292177 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:20.292146 5047 scope.go:122] "RemoveContainer" containerID="6bfa287be7a9eddd82860bbb2bb4a305fb81803b776a95b7c127f09a31a7f7f6" Oct 30 19:50:20.352222 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:20.352193 5047 scope.go:122] "RemoveContainer" containerID="7663aa77ca4c9b8e5823ae42f1ecb32daaf4303d1bde9a086a06e1b92f97baf1" Oct 30 19:50:20.352792 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:50:20.352758 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7663aa77ca4c9b8e5823ae42f1ecb32daaf4303d1bde9a086a06e1b92f97baf1\": container with ID starting with 7663aa77ca4c9b8e5823ae42f1ecb32daaf4303d1bde9a086a06e1b92f97baf1 not found: ID does not exist" containerID="7663aa77ca4c9b8e5823ae42f1ecb32daaf4303d1bde9a086a06e1b92f97baf1" Oct 30 19:50:20.352917 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:20.352802 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7663aa77ca4c9b8e5823ae42f1ecb32daaf4303d1bde9a086a06e1b92f97baf1"} err="failed to get container status \"7663aa77ca4c9b8e5823ae42f1ecb32daaf4303d1bde9a086a06e1b92f97baf1\": rpc error: code = NotFound desc = could not find container \"7663aa77ca4c9b8e5823ae42f1ecb32daaf4303d1bde9a086a06e1b92f97baf1\": container with ID starting with 7663aa77ca4c9b8e5823ae42f1ecb32daaf4303d1bde9a086a06e1b92f97baf1 not found: ID does not exist" Oct 30 19:50:20.352917 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:20.352827 5047 scope.go:122] "RemoveContainer" containerID="6bfa287be7a9eddd82860bbb2bb4a305fb81803b776a95b7c127f09a31a7f7f6" Oct 30 19:50:20.356220 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:50:20.356186 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bfa287be7a9eddd82860bbb2bb4a305fb81803b776a95b7c127f09a31a7f7f6\": container with ID starting with 6bfa287be7a9eddd82860bbb2bb4a305fb81803b776a95b7c127f09a31a7f7f6 not found: ID does not exist" containerID="6bfa287be7a9eddd82860bbb2bb4a305fb81803b776a95b7c127f09a31a7f7f6" Oct 30 19:50:20.356360 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:20.356227 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfa287be7a9eddd82860bbb2bb4a305fb81803b776a95b7c127f09a31a7f7f6"} err="failed to get container status \"6bfa287be7a9eddd82860bbb2bb4a305fb81803b776a95b7c127f09a31a7f7f6\": rpc error: code = NotFound desc = could not find container \"6bfa287be7a9eddd82860bbb2bb4a305fb81803b776a95b7c127f09a31a7f7f6\": container with ID starting with 6bfa287be7a9eddd82860bbb2bb4a305fb81803b776a95b7c127f09a31a7f7f6 not found: ID does not exist" Oct 30 19:50:20.362778 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:20.362751 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd"] Oct 30 19:50:20.370696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:20.370666 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-576a62-openshift-default-5b665b79f9-jrjwd"] Oct 30 19:50:21.050506 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:21.050472 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df7012d4-5d6b-4586-a4ac-629154c1d347" path="/var/lib/kubelet/pods/df7012d4-5d6b-4586-a4ac-629154c1d347/volumes" Oct 30 19:50:27.670594 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.670551 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx"] Oct 30 19:50:27.671557 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.671532 5047 cpu_manager.go:414] "containerMap: removing container" podUID="df7012d4-5d6b-4586-a4ac-629154c1d347" containerName="istio-proxy" Oct 30 19:50:27.671557 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.671557 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df7012d4-5d6b-4586-a4ac-629154c1d347" containerName="istio-proxy" Oct 30 19:50:27.671785 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.671603 5047 cpu_manager.go:414] "containerMap: removing container" podUID="df7012d4-5d6b-4586-a4ac-629154c1d347" containerName="istio-proxy" Oct 30 19:50:27.671785 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.671612 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df7012d4-5d6b-4586-a4ac-629154c1d347" containerName="istio-proxy" Oct 30 19:50:27.671891 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.671845 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="df7012d4-5d6b-4586-a4ac-629154c1d347" containerName="istio-proxy" Oct 30 19:50:27.671891 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.671867 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="df7012d4-5d6b-4586-a4ac-629154c1d347" containerName="istio-proxy" Oct 30 19:50:27.677458 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.677436 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.709668 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.709635 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx"] Oct 30 19:50:27.825617 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.825581 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f14230-b23c-42c4-9709-f67299e49c68-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.825789 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.825728 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.825789 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.825778 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f14230-b23c-42c4-9709-f67299e49c68-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.825889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.825866 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f14230-b23c-42c4-9709-f67299e49c68-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.825932 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.825916 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.825986 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.825973 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.826079 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.826064 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.826130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.826117 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.826214 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.826198 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4dh5\" (UniqueName: \"kubernetes.io/projected/e1f14230-b23c-42c4-9709-f67299e49c68-kube-api-access-d4dh5\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.927505 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.927411 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.927505 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.927492 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f14230-b23c-42c4-9709-f67299e49c68-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.927737 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.927542 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f14230-b23c-42c4-9709-f67299e49c68-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.927737 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.927579 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.927737 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.927605 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.927737 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.927656 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.927737 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.927722 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.928134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.927808 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-credential-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.928134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.927815 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4dh5\" (UniqueName: \"kubernetes.io/projected/e1f14230-b23c-42c4-9709-f67299e49c68-kube-api-access-d4dh5\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.928134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.927951 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f14230-b23c-42c4-9709-f67299e49c68-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.928134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.928066 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-workload-socket\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.928134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.928123 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-istio-data\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.928365 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.928130 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-workload-certs\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.928660 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.928638 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f14230-b23c-42c4-9709-f67299e49c68-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.930197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.930170 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-istio-envoy\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.930324 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.930205 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f14230-b23c-42c4-9709-f67299e49c68-istio-podinfo\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.944939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.944908 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f14230-b23c-42c4-9709-f67299e49c68-istio-token\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.945107 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.944993 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4dh5\" (UniqueName: \"kubernetes.io/projected/e1f14230-b23c-42c4-9709-f67299e49c68-kube-api-access-d4dh5\") pod \"e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:27.991408 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:27.991380 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:28.098722 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:28.097665 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:28.116603 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:28.116355 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:28.309001 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:28.308973 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx"] Oct 30 19:50:28.309738 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:50:28.309712 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f14230_b23c_42c4_9709_f67299e49c68.slice/crio-cc4a8aec2ee3f2e4d689ce936c9ac2f91005fe262405009541865ad8745c9207 WatchSource:0}: Error finding container cc4a8aec2ee3f2e4d689ce936c9ac2f91005fe262405009541865ad8745c9207: Status 404 returned error can't find the container with id cc4a8aec2ee3f2e4d689ce936c9ac2f91005fe262405009541865ad8745c9207 Oct 30 19:50:28.312634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:28.312597 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:50:28.312736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:28.312678 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:50:28.312736 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:28.312724 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:50:28.333536 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:28.333505 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" event={"ID":"e1f14230-b23c-42c4-9709-f67299e49c68","Type":"ContainerStarted","Data":"cc4a8aec2ee3f2e4d689ce936c9ac2f91005fe262405009541865ad8745c9207"} Oct 30 19:50:28.348999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:28.348971 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:50:29.370426 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:29.370380 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" event={"ID":"e1f14230-b23c-42c4-9709-f67299e49c68","Type":"ContainerStarted","Data":"afb8373164fc3546e76595177e3ab318047bc52a0103bae326a7a2aa0b077850"} Oct 30 19:50:29.427501 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:29.427451 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" podStartSLOduration=2.427438113 podStartE2EDuration="2.427438113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:29.417469294 +0000 UTC m=+2782.491924652" watchObservedRunningTime="2026-10-30 19:50:29.427438113 +0000 UTC m=+2782.501893495" Oct 30 19:50:29.992201 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:29.992169 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:29.994180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:29.994151 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 19:50:29.994319 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:29.994212 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" podUID="e1f14230-b23c-42c4-9709-f67299e49c68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 19:50:30.467704 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.467663 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck"] Oct 30 19:50:30.472651 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.472626 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 19:50:30.501772 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.499106 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-5a0a17-serving-cert\"" Oct 30 19:50:30.508086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.507145 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck"] Oct 30 19:50:30.591888 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.591855 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9958d25e-765d-45db-9d7d-a2210b1903c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck\" (UID: \"9958d25e-765d-45db-9d7d-a2210b1903c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 19:50:30.592123 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.591914 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9958d25e-765d-45db-9d7d-a2210b1903c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck\" (UID: \"9958d25e-765d-45db-9d7d-a2210b1903c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 19:50:30.592189 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.592157 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgfgx\" (UniqueName: \"kubernetes.io/projected/9958d25e-765d-45db-9d7d-a2210b1903c9-kube-api-access-vgfgx\") pod \"maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck\" (UID: \"9958d25e-765d-45db-9d7d-a2210b1903c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 19:50:30.592288 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.592267 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9958d25e-765d-45db-9d7d-a2210b1903c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck\" (UID: \"9958d25e-765d-45db-9d7d-a2210b1903c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 19:50:30.704030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.703902 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9958d25e-765d-45db-9d7d-a2210b1903c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck\" (UID: \"9958d25e-765d-45db-9d7d-a2210b1903c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 19:50:30.704030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.703989 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9958d25e-765d-45db-9d7d-a2210b1903c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck\" (UID: \"9958d25e-765d-45db-9d7d-a2210b1903c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 19:50:30.704293 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.704130 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgfgx\" (UniqueName: \"kubernetes.io/projected/9958d25e-765d-45db-9d7d-a2210b1903c9-kube-api-access-vgfgx\") pod \"maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck\" (UID: \"9958d25e-765d-45db-9d7d-a2210b1903c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 19:50:30.704293 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.704181 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9958d25e-765d-45db-9d7d-a2210b1903c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck\" (UID: \"9958d25e-765d-45db-9d7d-a2210b1903c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 19:50:30.715065 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.705838 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9958d25e-765d-45db-9d7d-a2210b1903c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck\" (UID: \"9958d25e-765d-45db-9d7d-a2210b1903c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 19:50:30.719399 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.719298 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9958d25e-765d-45db-9d7d-a2210b1903c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck\" (UID: \"9958d25e-765d-45db-9d7d-a2210b1903c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 19:50:30.733039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.732763 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9958d25e-765d-45db-9d7d-a2210b1903c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck\" (UID: \"9958d25e-765d-45db-9d7d-a2210b1903c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 19:50:30.759032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.756036 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgfgx\" (UniqueName: \"kubernetes.io/projected/9958d25e-765d-45db-9d7d-a2210b1903c9-kube-api-access-vgfgx\") pod \"maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck\" (UID: \"9958d25e-765d-45db-9d7d-a2210b1903c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 19:50:30.792829 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.791327 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 19:50:30.992521 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.992437 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 19:50:30.992521 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:30.992500 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" podUID="e1f14230-b23c-42c4-9709-f67299e49c68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 19:50:31.118830 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:31.118796 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck"] Oct 30 19:50:31.119130 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:50:31.119108 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9958d25e_765d_45db_9d7d_a2210b1903c9.slice/crio-f4b9b4b161c5be98e7bd4be210230e79c1fb2b3ff8063b611d83e7f729ac2d88 WatchSource:0}: Error finding container f4b9b4b161c5be98e7bd4be210230e79c1fb2b3ff8063b611d83e7f729ac2d88: Status 404 returned error can't find the container with id f4b9b4b161c5be98e7bd4be210230e79c1fb2b3ff8063b611d83e7f729ac2d88 Oct 30 19:50:31.411820 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:31.403850 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" event={"ID":"9958d25e-765d-45db-9d7d-a2210b1903c9","Type":"ContainerStarted","Data":"f4b9b4b161c5be98e7bd4be210230e79c1fb2b3ff8063b611d83e7f729ac2d88"} Oct 30 19:50:31.991618 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:31.991583 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 19:50:31.992039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:31.991635 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" podUID="e1f14230-b23c-42c4-9709-f67299e49c68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 19:50:32.416969 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:32.416934 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" event={"ID":"9958d25e-765d-45db-9d7d-a2210b1903c9","Type":"ContainerStarted","Data":"9c924bab0017e3561f9ba3b477a73e9126a48049794ac752e631afad2fb3a8c7"} Oct 30 19:50:32.417169 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:32.417070 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 19:50:32.465030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:32.464633 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" podStartSLOduration=2.020809061 podStartE2EDuration="2.464611036s" totalImagesPullingTime="443.801975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:32.453410199 +0000 UTC m=+2785.527865568" watchObservedRunningTime="2026-10-30 19:50:32.464611036 +0000 UTC m=+2785.539066396" Oct 30 19:50:32.992287 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:32.992251 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 19:50:32.992836 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:32.992325 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" podUID="e1f14230-b23c-42c4-9709-f67299e49c68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 19:50:33.992553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:33.992514 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 19:50:33.992963 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:33.992590 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" podUID="e1f14230-b23c-42c4-9709-f67299e49c68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 19:50:34.998116 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:34.998079 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:34.998530 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:34.998468 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:34.999177 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:34.999156 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 19:50:38.430631 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:38.430601 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 19:50:39.602872 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:50:39.602832 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:50:46.640845 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:46.640806 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4"] Oct 30 19:50:46.645789 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:46.645769 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" Oct 30 19:50:46.663498 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:46.663470 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4"] Oct 30 19:50:46.667706 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:46.667679 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-aae4d8\"" Oct 30 19:50:46.667885 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:46.667850 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-aae4d8-dockercfg-z947h\"" Oct 30 19:50:46.736756 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:46.736713 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w"] Oct 30 19:50:46.741533 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:46.741505 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" Oct 30 19:50:46.759675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:46.759641 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w"] Oct 30 19:50:46.778210 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:46.778175 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln2rq\" (UniqueName: \"kubernetes.io/projected/87185d96-e36e-47f7-8ea0-dd2e411b01ac-kube-api-access-ln2rq\") pod \"payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4\" (UID: \"87185d96-e36e-47f7-8ea0-dd2e411b01ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" Oct 30 19:50:46.778476 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:46.778453 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87185d96-e36e-47f7-8ea0-dd2e411b01ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4\" (UID: \"87185d96-e36e-47f7-8ea0-dd2e411b01ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" Oct 30 19:50:46.892725 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:46.892615 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb4cm\" (UniqueName: \"kubernetes.io/projected/c8d7d671-e339-41da-94ff-0c4b553a4147-kube-api-access-gb4cm\") pod \"payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w\" (UID: \"c8d7d671-e339-41da-94ff-0c4b553a4147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" Oct 30 19:50:46.893000 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:46.892836 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln2rq\" (UniqueName: \"kubernetes.io/projected/87185d96-e36e-47f7-8ea0-dd2e411b01ac-kube-api-access-ln2rq\") pod \"payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4\" (UID: \"87185d96-e36e-47f7-8ea0-dd2e411b01ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" Oct 30 19:50:46.893000 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:46.892889 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87185d96-e36e-47f7-8ea0-dd2e411b01ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4\" (UID: \"87185d96-e36e-47f7-8ea0-dd2e411b01ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" Oct 30 19:50:46.893000 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:46.892929 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d7d671-e339-41da-94ff-0c4b553a4147-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w\" (UID: \"c8d7d671-e339-41da-94ff-0c4b553a4147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" Oct 30 19:50:46.914029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:46.894353 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87185d96-e36e-47f7-8ea0-dd2e411b01ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4\" (UID: \"87185d96-e36e-47f7-8ea0-dd2e411b01ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" Oct 30 19:50:47.021075 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:47.017853 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w"] Oct 30 19:50:47.026762 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:47.026691 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d7d671-e339-41da-94ff-0c4b553a4147-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w\" (UID: \"c8d7d671-e339-41da-94ff-0c4b553a4147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" Oct 30 19:50:47.030642 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:47.027870 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d7d671-e339-41da-94ff-0c4b553a4147-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w\" (UID: \"c8d7d671-e339-41da-94ff-0c4b553a4147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" Oct 30 19:50:47.030884 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:47.029128 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln2rq\" (UniqueName: \"kubernetes.io/projected/87185d96-e36e-47f7-8ea0-dd2e411b01ac-kube-api-access-ln2rq\") pod \"payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4\" (UID: \"87185d96-e36e-47f7-8ea0-dd2e411b01ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" Oct 30 19:50:47.031550 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:47.031518 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb4cm\" (UniqueName: \"kubernetes.io/projected/c8d7d671-e339-41da-94ff-0c4b553a4147-kube-api-access-gb4cm\") pod \"payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w\" (UID: \"c8d7d671-e339-41da-94ff-0c4b553a4147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" Oct 30 19:50:47.192817 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:47.192739 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb4cm\" (UniqueName: \"kubernetes.io/projected/c8d7d671-e339-41da-94ff-0c4b553a4147-kube-api-access-gb4cm\") pod \"payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w\" (UID: \"c8d7d671-e339-41da-94ff-0c4b553a4147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" Oct 30 19:50:47.194832 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:47.194802 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" Oct 30 19:50:47.264103 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:47.264066 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" Oct 30 19:50:47.537290 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:50:47.537261 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d7d671_e339_41da_94ff_0c4b553a4147.slice/crio-33e5d7be6ca16f9bd67d231a0f2d727c26f1f35bced5213c8194800fffd5ef81 WatchSource:0}: Error finding container 33e5d7be6ca16f9bd67d231a0f2d727c26f1f35bced5213c8194800fffd5ef81: Status 404 returned error can't find the container with id 33e5d7be6ca16f9bd67d231a0f2d727c26f1f35bced5213c8194800fffd5ef81 Oct 30 19:50:47.537817 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:47.537795 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w"] Oct 30 19:50:47.577994 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:47.577964 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4"] Oct 30 19:50:47.578408 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:50:47.578380 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87185d96_e36e_47f7_8ea0_dd2e411b01ac.slice/crio-f38c9c69847c23d6c62c5086d56a68a88a9939d90dd371fc5dd0a8f1a67d62f2 WatchSource:0}: Error finding container f38c9c69847c23d6c62c5086d56a68a88a9939d90dd371fc5dd0a8f1a67d62f2: Status 404 returned error can't find the container with id f38c9c69847c23d6c62c5086d56a68a88a9939d90dd371fc5dd0a8f1a67d62f2 Oct 30 19:50:48.092316 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:48.092277 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn"] Oct 30 19:50:48.547264 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:48.547166 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" event={"ID":"c8d7d671-e339-41da-94ff-0c4b553a4147","Type":"ContainerStarted","Data":"a17cd27f75b0fba5a6e993271e547ad469a6dcee8dbd1a43096e5b01a79a39d5"} Oct 30 19:50:48.547264 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:48.547231 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" event={"ID":"c8d7d671-e339-41da-94ff-0c4b553a4147","Type":"ContainerStarted","Data":"33e5d7be6ca16f9bd67d231a0f2d727c26f1f35bced5213c8194800fffd5ef81"} Oct 30 19:50:48.548141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:48.548107 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" Oct 30 19:50:48.550838 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:48.550764 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" event={"ID":"87185d96-e36e-47f7-8ea0-dd2e411b01ac","Type":"ContainerStarted","Data":"cad4e96051190a8ce5801daaa629efb25eea155d8bfa93431a058a2079f5891b"} Oct 30 19:50:48.550838 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:48.550811 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" event={"ID":"87185d96-e36e-47f7-8ea0-dd2e411b01ac","Type":"ContainerStarted","Data":"f38c9c69847c23d6c62c5086d56a68a88a9939d90dd371fc5dd0a8f1a67d62f2"} Oct 30 19:50:48.551027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:48.550858 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" Oct 30 19:50:48.588653 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:48.588603 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" podStartSLOduration=2.588588722 podStartE2EDuration="2.588588722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:48.585763385 +0000 UTC m=+2801.660218744" watchObservedRunningTime="2026-10-30 19:50:48.588588722 +0000 UTC m=+2801.663044104" Oct 30 19:50:48.622300 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:48.622250 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" podStartSLOduration=2.6222343219999997 podStartE2EDuration="2.622234322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:48.620763791 +0000 UTC m=+2801.695219149" watchObservedRunningTime="2026-10-30 19:50:48.622234322 +0000 UTC m=+2801.696689715" Oct 30 19:50:59.587456 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:50:59.587426 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" Oct 30 19:51:00.578830 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:00.578797 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" Oct 30 19:51:02.259473 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.259431 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" podUID="9d7ca442-33fe-4785-ae1f-f48829771fc0" containerName="main" containerID="cri-o://288f830b6022fe8083beb1162e7e1aa1709d62570207686c1abd0d364fe252bf" gracePeriod=45 Oct 30 19:51:02.288420 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.288381 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" podUID="9d7ca442-33fe-4785-ae1f-f48829771fc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.178:8000/ready\": dial tcp 10.132.0.178:8000: connect: connection refused" Oct 30 19:51:02.616625 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.616592 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:51:02.807418 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.807305 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-home\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-home\") pod \"9d7ca442-33fe-4785-ae1f-f48829771fc0\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " Oct 30 19:51:02.807613 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.807474 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-model-cache\") pod \"9d7ca442-33fe-4785-ae1f-f48829771fc0\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " Oct 30 19:51:02.807613 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.807521 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-dshm\") pod \"9d7ca442-33fe-4785-ae1f-f48829771fc0\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " Oct 30 19:51:02.807744 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.807667 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-tmp-dir\") pod \"9d7ca442-33fe-4785-ae1f-f48829771fc0\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " Oct 30 19:51:02.808116 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.807806 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d7ca442-33fe-4785-ae1f-f48829771fc0-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d7ca442-33fe-4785-ae1f-f48829771fc0-tls-certs\") pod \"9d7ca442-33fe-4785-ae1f-f48829771fc0\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " Oct 30 19:51:02.808116 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.807808 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-model-cache" pod "9d7ca442-33fe-4785-ae1f-f48829771fc0" (UID: "9d7ca442-33fe-4785-ae1f-f48829771fc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:02.808116 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.807856 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-home" pod "9d7ca442-33fe-4785-ae1f-f48829771fc0" (UID: "9d7ca442-33fe-4785-ae1f-f48829771fc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:02.808116 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.807877 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7ca442-33fe-4785-ae1f-f48829771fc0-kube-api-access-bztwz\" (UniqueName: \"kubernetes.io/projected/9d7ca442-33fe-4785-ae1f-f48829771fc0-kube-api-access-bztwz\") pod \"9d7ca442-33fe-4785-ae1f-f48829771fc0\" (UID: \"9d7ca442-33fe-4785-ae1f-f48829771fc0\") " Oct 30 19:51:02.808116 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.808080 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-tmp-dir" pod "9d7ca442-33fe-4785-ae1f-f48829771fc0" (UID: "9d7ca442-33fe-4785-ae1f-f48829771fc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:02.808653 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.808627 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:51:02.808816 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.808656 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:51:02.808816 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.808672 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:51:02.810612 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.810583 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-dshm" pod "9d7ca442-33fe-4785-ae1f-f48829771fc0" (UID: "9d7ca442-33fe-4785-ae1f-f48829771fc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:02.810690 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.810608 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7ca442-33fe-4785-ae1f-f48829771fc0-kube-api-access-bztwz" pod "9d7ca442-33fe-4785-ae1f-f48829771fc0" (UID: "9d7ca442-33fe-4785-ae1f-f48829771fc0"). InnerVolumeSpecName "kube-api-access-bztwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:02.810690 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.810681 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d7ca442-33fe-4785-ae1f-f48829771fc0-tls-certs" pod "9d7ca442-33fe-4785-ae1f-f48829771fc0" (UID: "9d7ca442-33fe-4785-ae1f-f48829771fc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:02.854882 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.854848 5047 generic.go:356] "Generic (PLEG): container finished" podID="9d7ca442-33fe-4785-ae1f-f48829771fc0" containerID="288f830b6022fe8083beb1162e7e1aa1709d62570207686c1abd0d364fe252bf" exitCode=0 Oct 30 19:51:02.855085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.854895 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" Oct 30 19:51:02.855085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.854930 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" event={"ID":"9d7ca442-33fe-4785-ae1f-f48829771fc0","Type":"ContainerDied","Data":"288f830b6022fe8083beb1162e7e1aa1709d62570207686c1abd0d364fe252bf"} Oct 30 19:51:02.855085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.854977 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w" event={"ID":"9d7ca442-33fe-4785-ae1f-f48829771fc0","Type":"ContainerDied","Data":"0e0788fbfd89282f706c9a3da7849aba49bdea331829e9dcca5d46405c8059a0"} Oct 30 19:51:02.855085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.854999 5047 scope.go:122] "RemoveContainer" containerID="288f830b6022fe8083beb1162e7e1aa1709d62570207686c1abd0d364fe252bf" Oct 30 19:51:02.869786 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.869760 5047 scope.go:122] "RemoveContainer" containerID="288f830b6022fe8083beb1162e7e1aa1709d62570207686c1abd0d364fe252bf" Oct 30 19:51:02.870203 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:51:02.870179 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288f830b6022fe8083beb1162e7e1aa1709d62570207686c1abd0d364fe252bf\": container with ID starting with 288f830b6022fe8083beb1162e7e1aa1709d62570207686c1abd0d364fe252bf not found: ID does not exist" containerID="288f830b6022fe8083beb1162e7e1aa1709d62570207686c1abd0d364fe252bf" Oct 30 19:51:02.870306 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.870215 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288f830b6022fe8083beb1162e7e1aa1709d62570207686c1abd0d364fe252bf"} err="failed to get container status \"288f830b6022fe8083beb1162e7e1aa1709d62570207686c1abd0d364fe252bf\": rpc error: code = NotFound desc = could not find container \"288f830b6022fe8083beb1162e7e1aa1709d62570207686c1abd0d364fe252bf\": container with ID starting with 288f830b6022fe8083beb1162e7e1aa1709d62570207686c1abd0d364fe252bf not found: ID does not exist" Oct 30 19:51:02.904434 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.904399 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w"] Oct 30 19:51:02.910544 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.910507 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d7ca442-33fe-4785-ae1f-f48829771fc0-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:51:02.910544 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.910549 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bztwz\" (UniqueName: \"kubernetes.io/projected/9d7ca442-33fe-4785-ae1f-f48829771fc0-kube-api-access-bztwz\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:51:02.910780 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.910564 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d7ca442-33fe-4785-ae1f-f48829771fc0-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:51:02.910890 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:02.910862 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-095f55/auth-test-model-095f55-kserve-66c4ffd7f7-lkc8w"] Oct 30 19:51:03.053902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.053866 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d7ca442-33fe-4785-ae1f-f48829771fc0" path="/var/lib/kubelet/pods/9d7ca442-33fe-4785-ae1f-f48829771fc0/volumes" Oct 30 19:51:03.116246 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.116203 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" podUID="7735d239-4062-4651-84be-4ea42357260e" containerName="main" containerID="cri-o://d8d2b4a0ed546e00a3d8c449301002765ef07a8d6b98163d2424d0f813f3ba56" gracePeriod=45 Oct 30 19:51:03.453194 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.453160 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:51:03.520922 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.520870 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-model-cache\") pod \"7735d239-4062-4651-84be-4ea42357260e\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " Oct 30 19:51:03.521155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.520944 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-home\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-home\") pod \"7735d239-4062-4651-84be-4ea42357260e\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " Oct 30 19:51:03.521155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.521126 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-dshm\") pod \"7735d239-4062-4651-84be-4ea42357260e\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " Oct 30 19:51:03.521377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.521185 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-tmp-dir\") pod \"7735d239-4062-4651-84be-4ea42357260e\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " Oct 30 19:51:03.521377 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.521230 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7735d239-4062-4651-84be-4ea42357260e-tls-certs\" (UniqueName: \"kubernetes.io/secret/7735d239-4062-4651-84be-4ea42357260e-tls-certs\") pod \"7735d239-4062-4651-84be-4ea42357260e\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " Oct 30 19:51:03.521531 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.521507 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7735d239-4062-4651-84be-4ea42357260e-kube-api-access-bl4ct\" (UniqueName: \"kubernetes.io/projected/7735d239-4062-4651-84be-4ea42357260e-kube-api-access-bl4ct\") pod \"7735d239-4062-4651-84be-4ea42357260e\" (UID: \"7735d239-4062-4651-84be-4ea42357260e\") " Oct 30 19:51:03.522195 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.522162 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-tmp-dir" pod "7735d239-4062-4651-84be-4ea42357260e" (UID: "7735d239-4062-4651-84be-4ea42357260e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:03.524176 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.524145 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-model-cache" pod "7735d239-4062-4651-84be-4ea42357260e" (UID: "7735d239-4062-4651-84be-4ea42357260e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:03.524894 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.524868 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-home" pod "7735d239-4062-4651-84be-4ea42357260e" (UID: "7735d239-4062-4651-84be-4ea42357260e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:03.525185 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.525164 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:51:03.525311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.525297 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:51:03.525409 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.525398 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:51:03.527396 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.527366 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7735d239-4062-4651-84be-4ea42357260e-tls-certs" pod "7735d239-4062-4651-84be-4ea42357260e" (UID: "7735d239-4062-4651-84be-4ea42357260e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:03.527614 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.527573 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7735d239-4062-4651-84be-4ea42357260e-kube-api-access-bl4ct" pod "7735d239-4062-4651-84be-4ea42357260e" (UID: "7735d239-4062-4651-84be-4ea42357260e"). InnerVolumeSpecName "kube-api-access-bl4ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:03.529256 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.529227 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-dshm" pod "7735d239-4062-4651-84be-4ea42357260e" (UID: "7735d239-4062-4651-84be-4ea42357260e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:03.627094 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.627049 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl4ct\" (UniqueName: \"kubernetes.io/projected/7735d239-4062-4651-84be-4ea42357260e-kube-api-access-bl4ct\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:51:03.627094 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.627084 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7735d239-4062-4651-84be-4ea42357260e-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:51:03.627094 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.627095 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7735d239-4062-4651-84be-4ea42357260e-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:51:03.868542 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.868502 5047 generic.go:356] "Generic (PLEG): container finished" podID="7735d239-4062-4651-84be-4ea42357260e" containerID="d8d2b4a0ed546e00a3d8c449301002765ef07a8d6b98163d2424d0f813f3ba56" exitCode=0 Oct 30 19:51:03.868757 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.868577 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" Oct 30 19:51:03.868757 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.868591 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" event={"ID":"7735d239-4062-4651-84be-4ea42357260e","Type":"ContainerDied","Data":"d8d2b4a0ed546e00a3d8c449301002765ef07a8d6b98163d2424d0f813f3ba56"} Oct 30 19:51:03.868757 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.868639 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn" event={"ID":"7735d239-4062-4651-84be-4ea42357260e","Type":"ContainerDied","Data":"41a4bce122531e5b7262074d99c4767caaf84e716b705d3aedff215e4c773e67"} Oct 30 19:51:03.868757 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.868659 5047 scope.go:122] "RemoveContainer" containerID="d8d2b4a0ed546e00a3d8c449301002765ef07a8d6b98163d2424d0f813f3ba56" Oct 30 19:51:03.882140 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.882117 5047 scope.go:122] "RemoveContainer" containerID="d8d2b4a0ed546e00a3d8c449301002765ef07a8d6b98163d2424d0f813f3ba56" Oct 30 19:51:03.883948 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:51:03.883922 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d2b4a0ed546e00a3d8c449301002765ef07a8d6b98163d2424d0f813f3ba56\": container with ID starting with d8d2b4a0ed546e00a3d8c449301002765ef07a8d6b98163d2424d0f813f3ba56 not found: ID does not exist" containerID="d8d2b4a0ed546e00a3d8c449301002765ef07a8d6b98163d2424d0f813f3ba56" Oct 30 19:51:03.884068 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.883957 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d2b4a0ed546e00a3d8c449301002765ef07a8d6b98163d2424d0f813f3ba56"} err="failed to get container status \"d8d2b4a0ed546e00a3d8c449301002765ef07a8d6b98163d2424d0f813f3ba56\": rpc error: code = NotFound desc = could not find container \"d8d2b4a0ed546e00a3d8c449301002765ef07a8d6b98163d2424d0f813f3ba56\": container with ID starting with d8d2b4a0ed546e00a3d8c449301002765ef07a8d6b98163d2424d0f813f3ba56 not found: ID does not exist" Oct 30 19:51:03.916088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.916053 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn"] Oct 30 19:51:03.925378 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:03.925346 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-870193/auth-test-model-870193-kserve-75fbcd7c65-89smn"] Oct 30 19:51:04.879923 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:04.879893 5047 generic.go:356] "Generic (PLEG): container finished" podID="f3712046-b84a-45fd-b20f-49c8d68adc11" containerID="64ce6678e99049f8c234669acbc47faf6ef5c2089a4f8c4b5806331dd25ede7b" exitCode=0 Oct 30 19:51:04.881285 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:04.879968 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" event={"ID":"f3712046-b84a-45fd-b20f-49c8d68adc11","Type":"ContainerDied","Data":"64ce6678e99049f8c234669acbc47faf6ef5c2089a4f8c4b5806331dd25ede7b"} Oct 30 19:51:04.881285 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:04.880487 5047 scope.go:122] "RemoveContainer" containerID="64ce6678e99049f8c234669acbc47faf6ef5c2089a4f8c4b5806331dd25ede7b" Oct 30 19:51:04.882492 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:04.882463 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:04.882561 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:04.882534 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:04.882597 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:04.882565 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:05.073115 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:05.072054 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7735d239-4062-4651-84be-4ea42357260e" path="/var/lib/kubelet/pods/7735d239-4062-4651-84be-4ea42357260e/volumes" Oct 30 19:51:05.842031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:05.838112 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:51:05.889220 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:05.889186 5047 generic.go:356] "Generic (PLEG): container finished" podID="9a300de1-9957-43a4-a24c-a7186fe1ba64" containerID="196eef2e0a36f718f9ad15e2d0c83509829c6f11ed1c90ce834239f8fbfb797f" exitCode=0 Oct 30 19:51:05.889689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:05.889256 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" event={"ID":"9a300de1-9957-43a4-a24c-a7186fe1ba64","Type":"ContainerDied","Data":"196eef2e0a36f718f9ad15e2d0c83509829c6f11ed1c90ce834239f8fbfb797f"} Oct 30 19:51:05.889816 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:05.889798 5047 scope.go:122] "RemoveContainer" containerID="196eef2e0a36f718f9ad15e2d0c83509829c6f11ed1c90ce834239f8fbfb797f" Oct 30 19:51:05.891635 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:05.891613 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" event={"ID":"f3712046-b84a-45fd-b20f-49c8d68adc11","Type":"ContainerStarted","Data":"22f1f7b47c15892b2f08ceeebb45fb5202c38a23386482dc3aeb100db1ad4b0c"} Oct 30 19:51:05.891787 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:05.891760 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:05.891847 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:05.891834 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:05.891884 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:05.891869 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:06.115209 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:06.115124 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:51:06.116052 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:06.116018 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 19:51:06.116187 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:06.116076 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" podUID="f3712046-b84a-45fd-b20f-49c8d68adc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 19:51:06.902824 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:06.902789 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" event={"ID":"9a300de1-9957-43a4-a24c-a7186fe1ba64","Type":"ContainerStarted","Data":"6caa892bb571631653335c8e32f5eca17be821deef4db87a06a6283ec27659b2"} Oct 30 19:51:07.114747 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:07.114710 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 19:51:07.114921 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:07.114759 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" podUID="f3712046-b84a-45fd-b20f-49c8d68adc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 19:51:07.489443 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:07.489271 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:51:07.490466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:07.490417 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 19:51:07.490619 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:07.490514 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" podUID="9a300de1-9957-43a4-a24c-a7186fe1ba64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 19:51:08.114576 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:08.114539 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 19:51:08.114959 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:08.114590 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" podUID="f3712046-b84a-45fd-b20f-49c8d68adc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 19:51:08.275154 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:08.275113 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:51:08.489639 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:08.489544 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 19:51:08.489810 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:08.489638 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" podUID="9a300de1-9957-43a4-a24c-a7186fe1ba64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 19:51:09.120028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:09.118971 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 19:51:09.120028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:09.119059 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" podUID="f3712046-b84a-45fd-b20f-49c8d68adc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 19:51:09.489075 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:09.488579 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 19:51:09.489075 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:09.488646 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" podUID="9a300de1-9957-43a4-a24c-a7186fe1ba64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 19:51:09.611488 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:51:09.611453 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:51:10.114664 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:10.114608 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 19:51:10.114961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:10.114921 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" podUID="f3712046-b84a-45fd-b20f-49c8d68adc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 19:51:10.489133 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:10.489041 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 19:51:10.489133 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:10.489096 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" podUID="9a300de1-9957-43a4-a24c-a7186fe1ba64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 19:51:11.115570 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:11.115537 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 19:51:11.115767 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:11.115587 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" podUID="f3712046-b84a-45fd-b20f-49c8d68adc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 19:51:11.488638 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:11.488554 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 19:51:11.488638 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:11.488607 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" podUID="9a300de1-9957-43a4-a24c-a7186fe1ba64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 19:51:12.120271 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:12.120235 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:51:12.120796 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:12.120774 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:51:12.121563 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:12.121538 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 19:51:12.493495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:12.493420 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:51:12.494496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:12.494473 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:51:39.693340 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:51:39.693244 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:51:47.978191 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:47.978147 5047 patch_prober.go:29] interesting pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:47.978756 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:47.978222 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" podUID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:51.509611 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:51.509571 5047 generic.go:356] "Generic (PLEG): container finished" podID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" containerID="d3033c99f6d6304cf6d9b723014b787b4375f0c84c2a148a5945cee80f16fe82" exitCode=0 Oct 30 19:51:51.510080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:51.509624 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" event={"ID":"f5b67c1b-d500-4e75-affb-e748e37b0cfb","Type":"ContainerDied","Data":"d3033c99f6d6304cf6d9b723014b787b4375f0c84c2a148a5945cee80f16fe82"} Oct 30 19:51:51.510134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:51.510117 5047 scope.go:122] "RemoveContainer" containerID="d3033c99f6d6304cf6d9b723014b787b4375f0c84c2a148a5945cee80f16fe82" Oct 30 19:51:51.512055 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:51.512022 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:51.512125 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:51.512097 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:51.512173 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:51.512137 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:52.548562 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:52.540833 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" event={"ID":"f5b67c1b-d500-4e75-affb-e748e37b0cfb","Type":"ContainerStarted","Data":"ac9500c921bd6d33247a6bd3a311e30b30c0245b4f25ab9084e0e842ead83283"} Oct 30 19:51:52.556567 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:52.556523 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:51:52.556930 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:52.556840 5047 patch_prober.go:29] interesting pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 19:51:52.557145 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:52.557120 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" podUID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 19:51:53.559593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:53.559552 5047 patch_prober.go:29] interesting pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 19:51:53.560133 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:53.559637 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" podUID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 19:51:54.558030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:54.556448 5047 patch_prober.go:29] interesting pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 19:51:54.558030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:54.556525 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" podUID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 19:51:55.559206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:55.559165 5047 patch_prober.go:29] interesting pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 19:51:55.559724 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:55.559225 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" podUID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 19:51:56.554469 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:56.554426 5047 patch_prober.go:29] interesting pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 19:51:56.556204 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:56.555778 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" podUID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 19:51:57.553809 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:57.553763 5047 patch_prober.go:29] interesting pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 19:51:57.554237 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:57.553845 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" podUID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 19:51:58.558367 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.558331 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:51:58.558828 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.558806 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:51:58.559567 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.559547 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:51:58.629169 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.629122 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v"] Oct 30 19:51:58.629890 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.629876 5047 cpu_manager.go:414] "containerMap: removing container" podUID="7735d239-4062-4651-84be-4ea42357260e" containerName="main" Oct 30 19:51:58.629941 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.629892 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7735d239-4062-4651-84be-4ea42357260e" containerName="main" Oct 30 19:51:58.629941 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.629928 5047 cpu_manager.go:414] "containerMap: removing container" podUID="9d7ca442-33fe-4785-ae1f-f48829771fc0" containerName="main" Oct 30 19:51:58.629941 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.629933 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7ca442-33fe-4785-ae1f-f48829771fc0" containerName="main" Oct 30 19:51:58.630086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.630074 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7ca442-33fe-4785-ae1f-f48829771fc0" containerName="main" Oct 30 19:51:58.630125 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.630094 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="7735d239-4062-4651-84be-4ea42357260e" containerName="main" Oct 30 19:51:58.633059 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.633042 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.647507 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.647478 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-287ea2-openshift-default-dockercfg-7kw4v\"" Oct 30 19:51:58.658104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.658072 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v"] Oct 30 19:51:58.820329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.820243 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.820329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.820288 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.820532 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.820485 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.820624 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.820608 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.820671 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.820633 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.820671 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.820652 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtk2n\" (UniqueName: \"kubernetes.io/projected/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-kube-api-access-vtk2n\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.820785 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.820732 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.820821 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.820788 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.820862 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.820821 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.922568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.922524 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.922754 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.922627 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.922754 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.922648 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.922754 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.922671 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtk2n\" (UniqueName: \"kubernetes.io/projected/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-kube-api-access-vtk2n\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.922754 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.922722 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.922976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.922784 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.922976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.922820 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.922976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.922880 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.922976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.922912 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.923238 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.923157 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-workload-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.923298 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.923252 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-credential-socket\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.923298 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.923275 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-workload-certs\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.923417 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.923348 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-data\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.923838 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.923811 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istiod-ca-cert\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.925167 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.925146 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-envoy\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.925461 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.925441 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-podinfo\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.939980 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.939956 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-token\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.940338 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.940315 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtk2n\" (UniqueName: \"kubernetes.io/projected/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-kube-api-access-vtk2n\") pod \"e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:58.947044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:58.947023 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:51:59.260792 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:59.260750 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v"] Oct 30 19:51:59.261682 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:51:59.261648 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd19e5f_186b_4b69_b6b5_99e53d3e62a2.slice/crio-717d4ad6d4e27fd6187a598dd49e82f29d8b39b9d93509cb23b5fcc692add666 WatchSource:0}: Error finding container 717d4ad6d4e27fd6187a598dd49e82f29d8b39b9d93509cb23b5fcc692add666: Status 404 returned error can't find the container with id 717d4ad6d4e27fd6187a598dd49e82f29d8b39b9d93509cb23b5fcc692add666 Oct 30 19:51:59.264629 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:59.264597 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:59.264712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:59.264661 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:59.264712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:59.264692 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:59.662656 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:59.662618 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" event={"ID":"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2","Type":"ContainerStarted","Data":"b5417d430c6c552de60775d57dd57543b0e54c0ef46308cce92b83360ee62d45"} Oct 30 19:51:59.663146 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:59.662684 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" event={"ID":"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2","Type":"ContainerStarted","Data":"717d4ad6d4e27fd6187a598dd49e82f29d8b39b9d93509cb23b5fcc692add666"} Oct 30 19:51:59.699595 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:59.699522 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" podStartSLOduration=1.6995032970000001 podStartE2EDuration="1.699503297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:59.698414493 +0000 UTC m=+2872.772869864" watchObservedRunningTime="2026-10-30 19:51:59.699503297 +0000 UTC m=+2872.773958659" Oct 30 19:51:59.953080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:51:59.950451 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:52:00.961074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:00.956575 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 19:52:00.961074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:00.956643 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" podUID="3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 19:52:01.960165 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:01.960116 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 19:52:01.960393 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:01.960182 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" podUID="3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 19:52:02.956031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:02.955465 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 19:52:02.956031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:02.955608 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" podUID="3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 19:52:03.960156 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:03.960121 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 19:52:03.960645 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:03.960603 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" podUID="3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 19:52:04.960033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:04.958109 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 19:52:04.960033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:04.958176 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" podUID="3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 19:52:06.093501 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:06.093462 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:52:06.094785 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:06.094755 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:52:06.096194 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:06.096159 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:52:09.605103 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:52:09.605062 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:52:32.395157 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:32.395119 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9"] Oct 30 19:52:39.602662 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:52:39.602617 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:52:47.418653 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:47.418607 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" podUID="038833d2-c645-45ff-861f-64af9e82bae3" containerName="main" containerID="cri-o://57a983f32920588b01f8e294cdf18c13e4db672161a0b3d846f6264bce0bbae8" gracePeriod=45 Oct 30 19:52:48.033016 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.032880 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" podUID="038833d2-c645-45ff-861f-64af9e82bae3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.177:8000/ready\": dial tcp 10.132.0.177:8000: connect: connection refused" Oct 30 19:52:48.280575 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.280542 5047 generic.go:356] "Generic (PLEG): container finished" podID="038833d2-c645-45ff-861f-64af9e82bae3" containerID="57a983f32920588b01f8e294cdf18c13e4db672161a0b3d846f6264bce0bbae8" exitCode=0 Oct 30 19:52:48.280773 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.280611 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" event={"ID":"038833d2-c645-45ff-861f-64af9e82bae3","Type":"ContainerDied","Data":"57a983f32920588b01f8e294cdf18c13e4db672161a0b3d846f6264bce0bbae8"} Oct 30 19:52:48.280773 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.280650 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" event={"ID":"038833d2-c645-45ff-861f-64af9e82bae3","Type":"ContainerDied","Data":"de9265ec1ae455faf4cfdae48069e838c7c2a9594f829a5488dc29c8b757ef66"} Oct 30 19:52:48.280773 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.280664 5047 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de9265ec1ae455faf4cfdae48069e838c7c2a9594f829a5488dc29c8b757ef66" Oct 30 19:52:48.296985 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.296916 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:52:48.372262 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.372225 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-home\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-home\") pod \"038833d2-c645-45ff-861f-64af9e82bae3\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " Oct 30 19:52:48.372483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.372315 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-tmp-dir\") pod \"038833d2-c645-45ff-861f-64af9e82bae3\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " Oct 30 19:52:48.372483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.372367 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-dshm\") pod \"038833d2-c645-45ff-861f-64af9e82bae3\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " Oct 30 19:52:48.372483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.372389 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038833d2-c645-45ff-861f-64af9e82bae3-kube-api-access-jmbz5\" (UniqueName: \"kubernetes.io/projected/038833d2-c645-45ff-861f-64af9e82bae3-kube-api-access-jmbz5\") pod \"038833d2-c645-45ff-861f-64af9e82bae3\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " Oct 30 19:52:48.372669 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.372520 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-model-cache\") pod \"038833d2-c645-45ff-861f-64af9e82bae3\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " Oct 30 19:52:48.372669 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.372547 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-home" pod "038833d2-c645-45ff-861f-64af9e82bae3" (UID: "038833d2-c645-45ff-861f-64af9e82bae3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:48.372669 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.372606 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-tmp-dir" pod "038833d2-c645-45ff-861f-64af9e82bae3" (UID: "038833d2-c645-45ff-861f-64af9e82bae3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:48.372669 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.372633 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/038833d2-c645-45ff-861f-64af9e82bae3-tls-certs\" (UniqueName: \"kubernetes.io/secret/038833d2-c645-45ff-861f-64af9e82bae3-tls-certs\") pod \"038833d2-c645-45ff-861f-64af9e82bae3\" (UID: \"038833d2-c645-45ff-861f-64af9e82bae3\") " Oct 30 19:52:48.373106 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.373081 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-model-cache" pod "038833d2-c645-45ff-861f-64af9e82bae3" (UID: "038833d2-c645-45ff-861f-64af9e82bae3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:48.373306 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.373270 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:52:48.373306 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.373299 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:52:48.373482 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.373313 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:52:48.375135 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.375106 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038833d2-c645-45ff-861f-64af9e82bae3-kube-api-access-jmbz5" pod "038833d2-c645-45ff-861f-64af9e82bae3" (UID: "038833d2-c645-45ff-861f-64af9e82bae3"). InnerVolumeSpecName "kube-api-access-jmbz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:48.375241 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.375191 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-dshm" pod "038833d2-c645-45ff-861f-64af9e82bae3" (UID: "038833d2-c645-45ff-861f-64af9e82bae3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:48.375543 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.375522 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/038833d2-c645-45ff-861f-64af9e82bae3-tls-certs" pod "038833d2-c645-45ff-861f-64af9e82bae3" (UID: "038833d2-c645-45ff-861f-64af9e82bae3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:52:48.474407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.474366 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038833d2-c645-45ff-861f-64af9e82bae3-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:52:48.474407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.474401 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/038833d2-c645-45ff-861f-64af9e82bae3-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:52:48.474407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:48.474412 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmbz5\" (UniqueName: \"kubernetes.io/projected/038833d2-c645-45ff-861f-64af9e82bae3-kube-api-access-jmbz5\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:52:49.287685 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:49.287649 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9" Oct 30 19:52:49.335046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:49.334991 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9"] Oct 30 19:52:49.344950 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:49.344912 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a56cb5/rate-test-model-a56cb5-kserve-6487c48ddf-6g6d9"] Oct 30 19:52:51.051548 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:51.051509 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="038833d2-c645-45ff-861f-64af9e82bae3" path="/var/lib/kubelet/pods/038833d2-c645-45ff-861f-64af9e82bae3/volumes" Oct 30 19:52:54.402028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:52:54.401327 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz"] Oct 30 19:53:05.506085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.506050 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd"] Oct 30 19:53:05.506882 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.506862 5047 cpu_manager.go:414] "containerMap: removing container" podUID="038833d2-c645-45ff-861f-64af9e82bae3" containerName="main" Oct 30 19:53:05.506882 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.506881 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="038833d2-c645-45ff-861f-64af9e82bae3" containerName="main" Oct 30 19:53:05.507102 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.507089 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="038833d2-c645-45ff-861f-64af9e82bae3" containerName="main" Oct 30 19:53:05.515802 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.515774 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.534123 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.534063 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd"] Oct 30 19:53:05.542696 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.542666 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1b36ad-openshift-default-dockercfg-27tfc\"" Oct 30 19:53:05.620459 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.620420 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2547bf26-2d3b-4565-b16c-c37d17d410d7-istiod-ca-cert\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.620654 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.620469 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-podinfo\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.620654 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.620493 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-token\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.620743 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.620643 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-workload-socket\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.620743 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.620683 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-credential-socket\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.620743 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.620715 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67llt\" (UniqueName: \"kubernetes.io/projected/2547bf26-2d3b-4565-b16c-c37d17d410d7-kube-api-access-67llt\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.620863 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.620787 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-data\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.620900 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.620882 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-workload-certs\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.621056 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.621043 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-envoy\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.729028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.728380 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2547bf26-2d3b-4565-b16c-c37d17d410d7-istiod-ca-cert\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.729028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.728446 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-podinfo\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.729028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.728476 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-token\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.729028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.728549 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-workload-socket\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.729028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.728589 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-credential-socket\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.729028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.728617 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67llt\" (UniqueName: \"kubernetes.io/projected/2547bf26-2d3b-4565-b16c-c37d17d410d7-kube-api-access-67llt\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.729028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.728686 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-data\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.729028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.728796 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-workload-certs\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.729620 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.729069 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-envoy\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.737320 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.735446 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-workload-socket\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.751028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.745689 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2547bf26-2d3b-4565-b16c-c37d17d410d7-istiod-ca-cert\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.751028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.746556 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-credential-socket\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.751028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.746810 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-data\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.751028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.747064 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-workload-certs\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.771084 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.764905 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-podinfo\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.771679 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.771644 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-envoy\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.790449 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.790409 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-token\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.813053 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.812997 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67llt\" (UniqueName: \"kubernetes.io/projected/2547bf26-2d3b-4565-b16c-c37d17d410d7-kube-api-access-67llt\") pod \"e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:05.842364 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:05.842324 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:06.275378 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:06.275351 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd"] Oct 30 19:53:06.275987 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:53:06.275963 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2547bf26_2d3b_4565_b16c_c37d17d410d7.slice/crio-bf9301ce6ed1bba35d3c533ff524ce8b450f2279f2fa69c5c61d1bbb168adc2d WatchSource:0}: Error finding container bf9301ce6ed1bba35d3c533ff524ce8b450f2279f2fa69c5c61d1bbb168adc2d: Status 404 returned error can't find the container with id bf9301ce6ed1bba35d3c533ff524ce8b450f2279f2fa69c5c61d1bbb168adc2d Oct 30 19:53:06.278830 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:06.278794 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:06.278910 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:06.278864 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:06.278910 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:06.278892 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:06.431992 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:06.431955 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" event={"ID":"2547bf26-2d3b-4565-b16c-c37d17d410d7","Type":"ContainerStarted","Data":"b55a051a5b0d7c5e39a75bf8e4f9b5812b6845acf9cc2d890ed4a9cc885b843e"} Oct 30 19:53:06.432234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:06.432041 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" event={"ID":"2547bf26-2d3b-4565-b16c-c37d17d410d7","Type":"ContainerStarted","Data":"bf9301ce6ed1bba35d3c533ff524ce8b450f2279f2fa69c5c61d1bbb168adc2d"} Oct 30 19:53:06.470936 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:06.470882 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" podStartSLOduration=1.470864692 podStartE2EDuration="1.470864692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:06.469991707 +0000 UTC m=+2939.544447069" watchObservedRunningTime="2026-10-30 19:53:06.470864692 +0000 UTC m=+2939.545320051" Oct 30 19:53:06.845294 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:06.845255 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:07.851157 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:07.851124 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:08.447454 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:08.447415 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:08.448835 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:08.448813 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:53:09.438910 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.438858 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" podUID="f5dab94b-516a-4db7-9b19-e6f95456abfa" containerName="main" containerID="cri-o://997f45974169c22883bf57a9bf4c7adcece66e2dca25050c8bdf560ed14658a7" gracePeriod=45 Oct 30 19:53:09.605198 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:53:09.605161 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:53:09.718225 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.718197 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:53:09.813085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.813051 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-model-cache\") pod \"f5dab94b-516a-4db7-9b19-e6f95456abfa\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " Oct 30 19:53:09.813264 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.813101 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-home\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-home\") pod \"f5dab94b-516a-4db7-9b19-e6f95456abfa\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " Oct 30 19:53:09.813264 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.813154 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-tmp-dir\") pod \"f5dab94b-516a-4db7-9b19-e6f95456abfa\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " Oct 30 19:53:09.813385 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.813305 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5dab94b-516a-4db7-9b19-e6f95456abfa-kube-api-access-q8f55\" (UniqueName: \"kubernetes.io/projected/f5dab94b-516a-4db7-9b19-e6f95456abfa-kube-api-access-q8f55\") pod \"f5dab94b-516a-4db7-9b19-e6f95456abfa\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " Oct 30 19:53:09.813385 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.813362 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-model-cache" pod "f5dab94b-516a-4db7-9b19-e6f95456abfa" (UID: "f5dab94b-516a-4db7-9b19-e6f95456abfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:09.813385 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.813381 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-dshm\") pod \"f5dab94b-516a-4db7-9b19-e6f95456abfa\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " Oct 30 19:53:09.813567 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.813402 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-home" pod "f5dab94b-516a-4db7-9b19-e6f95456abfa" (UID: "f5dab94b-516a-4db7-9b19-e6f95456abfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:09.813567 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.813526 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5dab94b-516a-4db7-9b19-e6f95456abfa-tls-certs\" (UniqueName: \"kubernetes.io/secret/f5dab94b-516a-4db7-9b19-e6f95456abfa-tls-certs\") pod \"f5dab94b-516a-4db7-9b19-e6f95456abfa\" (UID: \"f5dab94b-516a-4db7-9b19-e6f95456abfa\") " Oct 30 19:53:09.813567 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.813523 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-tmp-dir" pod "f5dab94b-516a-4db7-9b19-e6f95456abfa" (UID: "f5dab94b-516a-4db7-9b19-e6f95456abfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:09.814117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.814094 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:53:09.814117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.814115 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:53:09.814117 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.814124 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:53:09.815683 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.815658 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5dab94b-516a-4db7-9b19-e6f95456abfa-tls-certs" pod "f5dab94b-516a-4db7-9b19-e6f95456abfa" (UID: "f5dab94b-516a-4db7-9b19-e6f95456abfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:09.815937 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.815921 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-dshm" pod "f5dab94b-516a-4db7-9b19-e6f95456abfa" (UID: "f5dab94b-516a-4db7-9b19-e6f95456abfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:09.816083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.816038 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5dab94b-516a-4db7-9b19-e6f95456abfa-kube-api-access-q8f55" pod "f5dab94b-516a-4db7-9b19-e6f95456abfa" (UID: "f5dab94b-516a-4db7-9b19-e6f95456abfa"). InnerVolumeSpecName "kube-api-access-q8f55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:09.915103 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.915071 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5dab94b-516a-4db7-9b19-e6f95456abfa-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:53:09.915103 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.915101 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8f55\" (UniqueName: \"kubernetes.io/projected/f5dab94b-516a-4db7-9b19-e6f95456abfa-kube-api-access-q8f55\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:53:09.915318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:09.915113 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5dab94b-516a-4db7-9b19-e6f95456abfa-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:53:10.463903 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.463865 5047 generic.go:356] "Generic (PLEG): container finished" podID="f5dab94b-516a-4db7-9b19-e6f95456abfa" containerID="997f45974169c22883bf57a9bf4c7adcece66e2dca25050c8bdf560ed14658a7" exitCode=0 Oct 30 19:53:10.464487 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.463913 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" Oct 30 19:53:10.464487 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.463949 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" event={"ID":"f5dab94b-516a-4db7-9b19-e6f95456abfa","Type":"ContainerDied","Data":"997f45974169c22883bf57a9bf4c7adcece66e2dca25050c8bdf560ed14658a7"} Oct 30 19:53:10.464487 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.463999 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz" event={"ID":"f5dab94b-516a-4db7-9b19-e6f95456abfa","Type":"ContainerDied","Data":"7c95b1c6f436c7339b2f5c97f04e1cf1f4147196e261b4003b9e78a4d748fdad"} Oct 30 19:53:10.464487 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.464026 5047 scope.go:122] "RemoveContainer" containerID="997f45974169c22883bf57a9bf4c7adcece66e2dca25050c8bdf560ed14658a7" Oct 30 19:53:10.478047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.476513 5047 scope.go:122] "RemoveContainer" containerID="997f45974169c22883bf57a9bf4c7adcece66e2dca25050c8bdf560ed14658a7" Oct 30 19:53:10.478047 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:53:10.477029 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997f45974169c22883bf57a9bf4c7adcece66e2dca25050c8bdf560ed14658a7\": container with ID starting with 997f45974169c22883bf57a9bf4c7adcece66e2dca25050c8bdf560ed14658a7 not found: ID does not exist" containerID="997f45974169c22883bf57a9bf4c7adcece66e2dca25050c8bdf560ed14658a7" Oct 30 19:53:10.478047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.477068 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997f45974169c22883bf57a9bf4c7adcece66e2dca25050c8bdf560ed14658a7"} err="failed to get container status \"997f45974169c22883bf57a9bf4c7adcece66e2dca25050c8bdf560ed14658a7\": rpc error: code = NotFound desc = could not find container \"997f45974169c22883bf57a9bf4c7adcece66e2dca25050c8bdf560ed14658a7\": container with ID starting with 997f45974169c22883bf57a9bf4c7adcece66e2dca25050c8bdf560ed14658a7 not found: ID does not exist" Oct 30 19:53:10.515150 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.515111 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz"] Oct 30 19:53:10.523323 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.523290 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-f83282/rate-test-model-f83282-kserve-756d87c59c-j2lqz"] Oct 30 19:53:10.769403 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.769321 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d"] Oct 30 19:53:10.769593 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.769551 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" podUID="ed7c056d-fb1d-49f8-9371-be32ecfdf069" containerName="payload-processing" containerID="cri-o://78fac3604b7086db0e0ffb53802d88fd974f76bc102dace7b5147fd569fee207" gracePeriod=30 Oct 30 19:53:10.833115 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.833080 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf"] Oct 30 19:53:10.835348 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.834402 5047 cpu_manager.go:414] "containerMap: removing container" podUID="f5dab94b-516a-4db7-9b19-e6f95456abfa" containerName="main" Oct 30 19:53:10.835348 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.834427 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5dab94b-516a-4db7-9b19-e6f95456abfa" containerName="main" Oct 30 19:53:10.835348 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.834652 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5dab94b-516a-4db7-9b19-e6f95456abfa" containerName="main" Oct 30 19:53:10.839916 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.839890 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" Oct 30 19:53:10.853231 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.853196 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf"] Oct 30 19:53:10.868547 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.868496 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1b36ad\"" Oct 30 19:53:10.868845 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.868824 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1b36ad-dockercfg-2fsms\"" Oct 30 19:53:10.894892 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.894843 5047 scope.go:122] "RemoveContainer" containerID="57a983f32920588b01f8e294cdf18c13e4db672161a0b3d846f6264bce0bbae8" Oct 30 19:53:10.933479 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.933446 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5744eb2c-c3d8-4ad7-8619-21075c182483-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf\" (UID: \"5744eb2c-c3d8-4ad7-8619-21075c182483\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" Oct 30 19:53:10.933662 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:10.933609 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zg5q\" (UniqueName: \"kubernetes.io/projected/5744eb2c-c3d8-4ad7-8619-21075c182483-kube-api-access-2zg5q\") pod \"payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf\" (UID: \"5744eb2c-c3d8-4ad7-8619-21075c182483\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" Oct 30 19:53:11.045454 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.034925 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zg5q\" (UniqueName: \"kubernetes.io/projected/5744eb2c-c3d8-4ad7-8619-21075c182483-kube-api-access-2zg5q\") pod \"payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf\" (UID: \"5744eb2c-c3d8-4ad7-8619-21075c182483\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" Oct 30 19:53:11.045454 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.035207 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5744eb2c-c3d8-4ad7-8619-21075c182483-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf\" (UID: \"5744eb2c-c3d8-4ad7-8619-21075c182483\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" Oct 30 19:53:11.045454 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.036234 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5744eb2c-c3d8-4ad7-8619-21075c182483-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf\" (UID: \"5744eb2c-c3d8-4ad7-8619-21075c182483\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" Oct 30 19:53:11.058381 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.058342 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zg5q\" (UniqueName: \"kubernetes.io/projected/5744eb2c-c3d8-4ad7-8619-21075c182483-kube-api-access-2zg5q\") pod \"payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf\" (UID: \"5744eb2c-c3d8-4ad7-8619-21075c182483\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" Oct 30 19:53:11.074808 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.074762 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5dab94b-516a-4db7-9b19-e6f95456abfa" path="/var/lib/kubelet/pods/f5dab94b-516a-4db7-9b19-e6f95456abfa/volumes" Oct 30 19:53:11.121337 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.121310 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" Oct 30 19:53:11.135730 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.135702 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed7c056d-fb1d-49f8-9371-be32ecfdf069-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7c056d-fb1d-49f8-9371-be32ecfdf069-plugins-config-volume\") pod \"ed7c056d-fb1d-49f8-9371-be32ecfdf069\" (UID: \"ed7c056d-fb1d-49f8-9371-be32ecfdf069\") " Oct 30 19:53:11.136079 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.136050 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7c056d-fb1d-49f8-9371-be32ecfdf069-kube-api-access-zczt7\" (UniqueName: \"kubernetes.io/projected/ed7c056d-fb1d-49f8-9371-be32ecfdf069-kube-api-access-zczt7\") pod \"ed7c056d-fb1d-49f8-9371-be32ecfdf069\" (UID: \"ed7c056d-fb1d-49f8-9371-be32ecfdf069\") " Oct 30 19:53:11.136181 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.136087 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7c056d-fb1d-49f8-9371-be32ecfdf069-plugins-config-volume" pod "ed7c056d-fb1d-49f8-9371-be32ecfdf069" (UID: "ed7c056d-fb1d-49f8-9371-be32ecfdf069"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:11.136935 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.136903 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7c056d-fb1d-49f8-9371-be32ecfdf069-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:53:11.138475 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.138449 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7c056d-fb1d-49f8-9371-be32ecfdf069-kube-api-access-zczt7" pod "ed7c056d-fb1d-49f8-9371-be32ecfdf069" (UID: "ed7c056d-fb1d-49f8-9371-be32ecfdf069"). InnerVolumeSpecName "kube-api-access-zczt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:11.189028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.184312 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" Oct 30 19:53:11.330123 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.318999 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zczt7\" (UniqueName: \"kubernetes.io/projected/ed7c056d-fb1d-49f8-9371-be32ecfdf069-kube-api-access-zczt7\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:53:11.502704 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.502672 5047 generic.go:356] "Generic (PLEG): container finished" podID="ed7c056d-fb1d-49f8-9371-be32ecfdf069" containerID="78fac3604b7086db0e0ffb53802d88fd974f76bc102dace7b5147fd569fee207" exitCode=0 Oct 30 19:53:11.503212 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.502742 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" Oct 30 19:53:11.503212 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.502764 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" event={"ID":"ed7c056d-fb1d-49f8-9371-be32ecfdf069","Type":"ContainerDied","Data":"78fac3604b7086db0e0ffb53802d88fd974f76bc102dace7b5147fd569fee207"} Oct 30 19:53:11.503212 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.502852 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d" event={"ID":"ed7c056d-fb1d-49f8-9371-be32ecfdf069","Type":"ContainerDied","Data":"b71b792c0946ab3798e6dc968237c8b669065d83617a62b1cd5e8f032de921b3"} Oct 30 19:53:11.503212 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.502914 5047 scope.go:122] "RemoveContainer" containerID="78fac3604b7086db0e0ffb53802d88fd974f76bc102dace7b5147fd569fee207" Oct 30 19:53:11.523659 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.523627 5047 scope.go:122] "RemoveContainer" containerID="78fac3604b7086db0e0ffb53802d88fd974f76bc102dace7b5147fd569fee207" Oct 30 19:53:11.524070 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:53:11.524044 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78fac3604b7086db0e0ffb53802d88fd974f76bc102dace7b5147fd569fee207\": container with ID starting with 78fac3604b7086db0e0ffb53802d88fd974f76bc102dace7b5147fd569fee207 not found: ID does not exist" containerID="78fac3604b7086db0e0ffb53802d88fd974f76bc102dace7b5147fd569fee207" Oct 30 19:53:11.524134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.524080 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78fac3604b7086db0e0ffb53802d88fd974f76bc102dace7b5147fd569fee207"} err="failed to get container status \"78fac3604b7086db0e0ffb53802d88fd974f76bc102dace7b5147fd569fee207\": rpc error: code = NotFound desc = could not find container \"78fac3604b7086db0e0ffb53802d88fd974f76bc102dace7b5147fd569fee207\": container with ID starting with 78fac3604b7086db0e0ffb53802d88fd974f76bc102dace7b5147fd569fee207 not found: ID does not exist" Oct 30 19:53:11.561817 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.561778 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d"] Oct 30 19:53:11.570112 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.570078 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54086b-865c77b484-vkr6d"] Oct 30 19:53:11.818245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:11.818210 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf"] Oct 30 19:53:11.820035 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:53:11.819989 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5744eb2c_c3d8_4ad7_8619_21075c182483.slice/crio-b25fa6855efe2733769909e3f4e226f3557bd50389bccd7c8be0801a574b8da9 WatchSource:0}: Error finding container b25fa6855efe2733769909e3f4e226f3557bd50389bccd7c8be0801a574b8da9: Status 404 returned error can't find the container with id b25fa6855efe2733769909e3f4e226f3557bd50389bccd7c8be0801a574b8da9 Oct 30 19:53:12.654030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:12.652553 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" event={"ID":"5744eb2c-c3d8-4ad7-8619-21075c182483","Type":"ContainerStarted","Data":"7985e863553b000438abf7abd280a4622afb40f77b99fb25b5d715bce78fdd2b"} Oct 30 19:53:12.654030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:12.652678 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" event={"ID":"5744eb2c-c3d8-4ad7-8619-21075c182483","Type":"ContainerStarted","Data":"b25fa6855efe2733769909e3f4e226f3557bd50389bccd7c8be0801a574b8da9"} Oct 30 19:53:12.654756 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:12.654069 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" Oct 30 19:53:12.732073 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:12.705871 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" podStartSLOduration=2.705851071 podStartE2EDuration="2.705851071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:12.703162878 +0000 UTC m=+2945.777618261" watchObservedRunningTime="2026-10-30 19:53:12.705851071 +0000 UTC m=+2945.780306431" Oct 30 19:53:13.058631 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:13.058521 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed7c056d-fb1d-49f8-9371-be32ecfdf069" path="/var/lib/kubelet/pods/ed7c056d-fb1d-49f8-9371-be32ecfdf069/volumes" Oct 30 19:53:24.788927 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:24.788897 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" Oct 30 19:53:31.548378 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.548331 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk"] Oct 30 19:53:31.549947 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.549917 5047 cpu_manager.go:414] "containerMap: removing container" podUID="ed7c056d-fb1d-49f8-9371-be32ecfdf069" containerName="payload-processing" Oct 30 19:53:31.549947 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.549942 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed7c056d-fb1d-49f8-9371-be32ecfdf069" containerName="payload-processing" Oct 30 19:53:31.550160 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.550148 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed7c056d-fb1d-49f8-9371-be32ecfdf069" containerName="payload-processing" Oct 30 19:53:31.555598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.555563 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.582260 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.582216 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk"] Oct 30 19:53:31.586687 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.586647 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-aae4d8\"/\"kube-root-ca.crt\"" Oct 30 19:53:31.586687 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.586667 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-aae4d8\"/\"auth-test-model-aae4d8-kserve-self-signed-certs\"" Oct 30 19:53:31.586687 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.586667 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-aae4d8\"/\"openshift-service-ca.crt\"" Oct 30 19:53:31.586987 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.586647 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-aae4d8\"/\"default-dockercfg-rnzcx\"" Oct 30 19:53:31.610325 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.610288 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-home\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.610504 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.610353 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-model-cache\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.610504 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.610449 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-dshm\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.610631 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.610528 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-tmp-dir\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.610631 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.610555 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dczt5\" (UniqueName: \"kubernetes.io/projected/37636069-645a-4941-97fd-a2fa4b3df247-kube-api-access-dczt5\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.610631 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.610585 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37636069-645a-4941-97fd-a2fa4b3df247-tls-certs\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.712661 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.712605 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-home\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.712889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.712681 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-model-cache\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.712889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.712721 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-dshm\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.712889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.712790 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-tmp-dir\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.712889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.712820 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dczt5\" (UniqueName: \"kubernetes.io/projected/37636069-645a-4941-97fd-a2fa4b3df247-kube-api-access-dczt5\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.712889 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.712856 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37636069-645a-4941-97fd-a2fa4b3df247-tls-certs\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.713244 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.713149 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-home\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.713313 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.713230 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-model-cache\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.713313 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.713260 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-tmp-dir\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.715165 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.715142 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-dshm\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.715500 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.715481 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37636069-645a-4941-97fd-a2fa4b3df247-tls-certs\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.730253 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.730221 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dczt5\" (UniqueName: \"kubernetes.io/projected/37636069-645a-4941-97fd-a2fa4b3df247-kube-api-access-dczt5\") pod \"auth-test-model-aae4d8-kserve-bcc875f57-8trgk\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:31.899488 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:31.889370 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:32.241598 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:32.241564 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk"] Oct 30 19:53:32.242953 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:53:32.242919 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37636069_645a_4941_97fd_a2fa4b3df247.slice/crio-edb360dcad788c628c8d070f846e6b5e932cf470245ce6682674f346ba11e256 WatchSource:0}: Error finding container edb360dcad788c628c8d070f846e6b5e932cf470245ce6682674f346ba11e256: Status 404 returned error can't find the container with id edb360dcad788c628c8d070f846e6b5e932cf470245ce6682674f346ba11e256 Oct 30 19:53:32.997241 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:32.997198 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" event={"ID":"37636069-645a-4941-97fd-a2fa4b3df247","Type":"ContainerStarted","Data":"21da91fc81391ec2f4951137c4c70e3e3d87acb04b5cd59578a05e94f97cb6a2"} Oct 30 19:53:32.997241 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:32.997249 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" event={"ID":"37636069-645a-4941-97fd-a2fa4b3df247","Type":"ContainerStarted","Data":"edb360dcad788c628c8d070f846e6b5e932cf470245ce6682674f346ba11e256"} Oct 30 19:53:33.035100 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:33.035037 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" podStartSLOduration=2.03502011 podStartE2EDuration="2.03502011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:33.031544765 +0000 UTC m=+2966.106000125" watchObservedRunningTime="2026-10-30 19:53:33.03502011 +0000 UTC m=+2966.109475471" Oct 30 19:53:33.889943 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:33.889900 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:38.276958 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:38.276922 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj"] Oct 30 19:53:38.277467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:38.277190 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" podUID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" containerName="istio-proxy" containerID="cri-o://50221a7adee5a4075732d0f4760c83dc87c0f34951cdea7f3c1703fdde09067d" gracePeriod=30 Oct 30 19:53:39.602679 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:53:39.602644 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:53:41.890096 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:41.890047 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:41.903170 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:41.903135 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:42.084755 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:42.084719 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:53:42.872317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:42.872283 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:53:42.872496 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:42.872366 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" podUID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:53:43.573967 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.573940 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:53:43.627661 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.627626 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istiod-ca-cert\") pod \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " Oct 30 19:53:43.627841 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.627750 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-workload-socket\") pod \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " Oct 30 19:53:43.627841 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.627797 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-envoy\") pod \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " Oct 30 19:53:43.627976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.627876 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-workload-certs\") pod \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " Oct 30 19:53:43.627976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.627934 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad094cfc-29f5-44cc-82b8-162680c6d3a9-kube-api-access-v774l\" (UniqueName: \"kubernetes.io/projected/ad094cfc-29f5-44cc-82b8-162680c6d3a9-kube-api-access-v774l\") pod \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " Oct 30 19:53:43.627976 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.627971 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-token\" (UniqueName: \"kubernetes.io/projected/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-token\") pod \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " Oct 30 19:53:43.628196 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.628057 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istiod-ca-cert" pod "ad094cfc-29f5-44cc-82b8-162680c6d3a9" (UID: "ad094cfc-29f5-44cc-82b8-162680c6d3a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:43.628196 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.628082 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-credential-socket\") pod \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " Oct 30 19:53:43.628196 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.628115 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-data\") pod \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " Oct 30 19:53:43.628196 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.628170 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-podinfo\") pod \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\" (UID: \"ad094cfc-29f5-44cc-82b8-162680c6d3a9\") " Oct 30 19:53:43.628408 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.628247 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-workload-certs" pod "ad094cfc-29f5-44cc-82b8-162680c6d3a9" (UID: "ad094cfc-29f5-44cc-82b8-162680c6d3a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:43.628469 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.628412 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-credential-socket" pod "ad094cfc-29f5-44cc-82b8-162680c6d3a9" (UID: "ad094cfc-29f5-44cc-82b8-162680c6d3a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:43.628469 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.628428 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-workload-socket" pod "ad094cfc-29f5-44cc-82b8-162680c6d3a9" (UID: "ad094cfc-29f5-44cc-82b8-162680c6d3a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:43.628583 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.628471 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-data" pod "ad094cfc-29f5-44cc-82b8-162680c6d3a9" (UID: "ad094cfc-29f5-44cc-82b8-162680c6d3a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:43.628868 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.628849 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:53:43.628955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.628873 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:53:43.628955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.628886 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:53:43.628955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.628901 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:53:43.628955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.628915 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:53:43.630823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.630798 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad094cfc-29f5-44cc-82b8-162680c6d3a9-kube-api-access-v774l" pod "ad094cfc-29f5-44cc-82b8-162680c6d3a9" (UID: "ad094cfc-29f5-44cc-82b8-162680c6d3a9"). InnerVolumeSpecName "kube-api-access-v774l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:43.630930 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.630806 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-podinfo" pod "ad094cfc-29f5-44cc-82b8-162680c6d3a9" (UID: "ad094cfc-29f5-44cc-82b8-162680c6d3a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:53:43.630930 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.630846 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-token" pod "ad094cfc-29f5-44cc-82b8-162680c6d3a9" (UID: "ad094cfc-29f5-44cc-82b8-162680c6d3a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:43.632874 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.632844 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-envoy" pod "ad094cfc-29f5-44cc-82b8-162680c6d3a9" (UID: "ad094cfc-29f5-44cc-82b8-162680c6d3a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:43.730338 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.730299 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:53:43.730338 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.730332 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v774l\" (UniqueName: \"kubernetes.io/projected/ad094cfc-29f5-44cc-82b8-162680c6d3a9-kube-api-access-v774l\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:53:43.730553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.730348 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:53:43.730553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:43.730361 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad094cfc-29f5-44cc-82b8-162680c6d3a9-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:53:44.093616 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:44.093580 5047 generic.go:356] "Generic (PLEG): container finished" podID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" containerID="50221a7adee5a4075732d0f4760c83dc87c0f34951cdea7f3c1703fdde09067d" exitCode=0 Oct 30 19:53:44.093828 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:44.093638 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" Oct 30 19:53:44.093828 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:44.093668 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" event={"ID":"ad094cfc-29f5-44cc-82b8-162680c6d3a9","Type":"ContainerDied","Data":"50221a7adee5a4075732d0f4760c83dc87c0f34951cdea7f3c1703fdde09067d"} Oct 30 19:53:44.093828 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:44.093733 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj" event={"ID":"ad094cfc-29f5-44cc-82b8-162680c6d3a9","Type":"ContainerDied","Data":"a2e5d1f3515d0741a020770a65f379f595b8095c1643973fde4a9f03073c8609"} Oct 30 19:53:44.093828 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:44.093755 5047 scope.go:122] "RemoveContainer" containerID="50221a7adee5a4075732d0f4760c83dc87c0f34951cdea7f3c1703fdde09067d" Oct 30 19:53:44.106695 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:44.106674 5047 scope.go:122] "RemoveContainer" containerID="37a8b3c2cdc3117e3568cf8c2b4d5041461096b417ae2d5356e70d2615c205c9" Oct 30 19:53:44.117679 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:44.117655 5047 scope.go:122] "RemoveContainer" containerID="50221a7adee5a4075732d0f4760c83dc87c0f34951cdea7f3c1703fdde09067d" Oct 30 19:53:44.117965 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:53:44.117945 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50221a7adee5a4075732d0f4760c83dc87c0f34951cdea7f3c1703fdde09067d\": container with ID starting with 50221a7adee5a4075732d0f4760c83dc87c0f34951cdea7f3c1703fdde09067d not found: ID does not exist" containerID="50221a7adee5a4075732d0f4760c83dc87c0f34951cdea7f3c1703fdde09067d" Oct 30 19:53:44.118037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:44.117975 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50221a7adee5a4075732d0f4760c83dc87c0f34951cdea7f3c1703fdde09067d"} err="failed to get container status \"50221a7adee5a4075732d0f4760c83dc87c0f34951cdea7f3c1703fdde09067d\": rpc error: code = NotFound desc = could not find container \"50221a7adee5a4075732d0f4760c83dc87c0f34951cdea7f3c1703fdde09067d\": container with ID starting with 50221a7adee5a4075732d0f4760c83dc87c0f34951cdea7f3c1703fdde09067d not found: ID does not exist" Oct 30 19:53:44.118037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:44.117992 5047 scope.go:122] "RemoveContainer" containerID="37a8b3c2cdc3117e3568cf8c2b4d5041461096b417ae2d5356e70d2615c205c9" Oct 30 19:53:44.118296 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:53:44.118276 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a8b3c2cdc3117e3568cf8c2b4d5041461096b417ae2d5356e70d2615c205c9\": container with ID starting with 37a8b3c2cdc3117e3568cf8c2b4d5041461096b417ae2d5356e70d2615c205c9 not found: ID does not exist" containerID="37a8b3c2cdc3117e3568cf8c2b4d5041461096b417ae2d5356e70d2615c205c9" Oct 30 19:53:44.118341 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:44.118304 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a8b3c2cdc3117e3568cf8c2b4d5041461096b417ae2d5356e70d2615c205c9"} err="failed to get container status \"37a8b3c2cdc3117e3568cf8c2b4d5041461096b417ae2d5356e70d2615c205c9\": rpc error: code = NotFound desc = could not find container \"37a8b3c2cdc3117e3568cf8c2b4d5041461096b417ae2d5356e70d2615c205c9\": container with ID starting with 37a8b3c2cdc3117e3568cf8c2b4d5041461096b417ae2d5356e70d2615c205c9 not found: ID does not exist" Oct 30 19:53:44.207666 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:44.207619 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj"] Oct 30 19:53:44.213494 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:44.213463 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-54086b-openshift-default-66b75895f7-q85nj"] Oct 30 19:53:45.057344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:45.057298 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" path="/var/lib/kubelet/pods/ad094cfc-29f5-44cc-82b8-162680c6d3a9/volumes" Oct 30 19:53:52.574612 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.574574 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx"] Oct 30 19:53:52.575425 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.575407 5047 cpu_manager.go:414] "containerMap: removing container" podUID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" containerName="istio-proxy" Oct 30 19:53:52.575480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.575427 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" containerName="istio-proxy" Oct 30 19:53:52.575480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.575465 5047 cpu_manager.go:414] "containerMap: removing container" podUID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" containerName="istio-proxy" Oct 30 19:53:52.575480 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.575470 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" containerName="istio-proxy" Oct 30 19:53:52.575596 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.575587 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" containerName="istio-proxy" Oct 30 19:53:52.580922 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.580901 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:53:52.597984 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.597949 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx"] Oct 30 19:53:52.602752 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.602714 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-1b36ad-serving-cert\"" Oct 30 19:53:52.719271 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.719219 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a7dbb86-134e-482d-b8f1-17a5105fdca6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx\" (UID: \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:53:52.719271 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.719275 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7dbb86-134e-482d-b8f1-17a5105fdca6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx\" (UID: \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:53:52.719534 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.719415 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a7dbb86-134e-482d-b8f1-17a5105fdca6-metrics-certs\") pod \"maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx\" (UID: \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:53:52.719599 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.719556 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r76jg\" (UniqueName: \"kubernetes.io/projected/3a7dbb86-134e-482d-b8f1-17a5105fdca6-kube-api-access-r76jg\") pod \"maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx\" (UID: \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:53:52.821406 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.821367 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a7dbb86-134e-482d-b8f1-17a5105fdca6-metrics-certs\") pod \"maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx\" (UID: \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:53:52.821613 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.821487 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r76jg\" (UniqueName: \"kubernetes.io/projected/3a7dbb86-134e-482d-b8f1-17a5105fdca6-kube-api-access-r76jg\") pod \"maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx\" (UID: \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:53:52.821708 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.821689 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a7dbb86-134e-482d-b8f1-17a5105fdca6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx\" (UID: \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:53:52.821783 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.821724 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7dbb86-134e-482d-b8f1-17a5105fdca6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx\" (UID: \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:53:52.822475 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.822451 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7dbb86-134e-482d-b8f1-17a5105fdca6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx\" (UID: \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:53:52.823967 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.823935 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a7dbb86-134e-482d-b8f1-17a5105fdca6-metrics-certs\") pod \"maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx\" (UID: \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:53:52.824132 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.824101 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a7dbb86-134e-482d-b8f1-17a5105fdca6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx\" (UID: \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:53:52.841971 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.841937 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r76jg\" (UniqueName: \"kubernetes.io/projected/3a7dbb86-134e-482d-b8f1-17a5105fdca6-kube-api-access-r76jg\") pod \"maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx\" (UID: \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:53:52.908033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:52.900352 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:53:53.242963 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:53:53.242927 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7dbb86_134e_482d_b8f1_17a5105fdca6.slice/crio-ab38099f695589e3d5a7ed9bf384307f746b47bb73633740d62974db96bea88b WatchSource:0}: Error finding container ab38099f695589e3d5a7ed9bf384307f746b47bb73633740d62974db96bea88b: Status 404 returned error can't find the container with id ab38099f695589e3d5a7ed9bf384307f746b47bb73633740d62974db96bea88b Oct 30 19:53:53.243870 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:53.243841 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx"] Oct 30 19:53:54.177788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:54.177727 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" event={"ID":"3a7dbb86-134e-482d-b8f1-17a5105fdca6","Type":"ContainerStarted","Data":"8f53add73de105a864a9bcd6d0280a929c3b8bdcbf820b6b2c19835544ae34a2"} Oct 30 19:53:54.177788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:54.177796 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" event={"ID":"3a7dbb86-134e-482d-b8f1-17a5105fdca6","Type":"ContainerStarted","Data":"ab38099f695589e3d5a7ed9bf384307f746b47bb73633740d62974db96bea88b"} Oct 30 19:53:54.178739 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:54.178061 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:53:54.216787 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:54.216725 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" podStartSLOduration=1.840058405 podStartE2EDuration="2.21670811s" totalImagesPullingTime="376.649705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:54.212399752 +0000 UTC m=+2987.286855104" watchObservedRunningTime="2026-10-30 19:53:54.21670811 +0000 UTC m=+2987.291163490" Oct 30 19:53:57.140024 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:53:57.139966 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q"] Oct 30 19:54:00.192266 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:00.192236 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:54:07.900982 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:07.900948 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:54:07.905263 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:07.905236 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:54:09.604962 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:54:09.604926 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:54:10.326818 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.326782 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz"] Oct 30 19:54:10.327860 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.327835 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad094cfc-29f5-44cc-82b8-162680c6d3a9" containerName="istio-proxy" Oct 30 19:54:10.332000 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.331979 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.334108 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.334080 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-podinfo\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.334308 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.334290 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcqhl\" (UniqueName: \"kubernetes.io/projected/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-kube-api-access-bcqhl\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.334486 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.334472 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-workload-certs\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.334615 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.334601 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istiod-ca-cert\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.334724 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.334711 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-token\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.334861 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.334845 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-workload-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.334979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.334966 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-data\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.335193 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.335178 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-credential-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.335527 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.335506 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-envoy\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.349971 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.349946 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-bf314f-openshift-default-dockercfg-sjttg\"" Oct 30 19:54:10.364332 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.364301 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz"] Oct 30 19:54:10.453333 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.453290 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-envoy\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.454703 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.454674 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-podinfo\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.456928 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.456902 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcqhl\" (UniqueName: \"kubernetes.io/projected/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-kube-api-access-bcqhl\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.475286 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.475243 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-workload-certs\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.476055 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.476031 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istiod-ca-cert\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.544815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.544785 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-token\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.545183 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.545158 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-workload-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.545345 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.545329 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-data\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.545474 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.545460 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-credential-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.546060 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.546034 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-credential-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.546210 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.493099 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istiod-ca-cert\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.546582 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.475924 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-workload-certs\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.546995 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.546973 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-workload-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.547913 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.547853 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-data\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.593081 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.558514 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcqhl\" (UniqueName: \"kubernetes.io/projected/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-kube-api-access-bcqhl\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.603428 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.603391 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-podinfo\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.604078 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.604027 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-envoy\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.645250 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.614980 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-token\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:10.681214 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:10.681169 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:11.029780 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:54:11.029747 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e3fe64_b5fe_453a_ae2b_7dec205c7931.slice/crio-d051af7d6508c695cb28214fe3b075796fbd5aa8fdbc7b2099cfc5317ed1721f WatchSource:0}: Error finding container d051af7d6508c695cb28214fe3b075796fbd5aa8fdbc7b2099cfc5317ed1721f: Status 404 returned error can't find the container with id d051af7d6508c695cb28214fe3b075796fbd5aa8fdbc7b2099cfc5317ed1721f Oct 30 19:54:11.029885 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:11.029791 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz"] Oct 30 19:54:11.033187 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:11.033155 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:11.033272 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:11.033226 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:11.033272 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:11.033261 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:11.314610 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:11.314513 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" event={"ID":"c4e3fe64-b5fe-453a-ae2b-7dec205c7931","Type":"ContainerStarted","Data":"1c2ff8d309070abb77eff14bdeabca03ab01bc4f16059298cba917c96dfc8659"} Oct 30 19:54:11.314610 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:11.314578 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" event={"ID":"c4e3fe64-b5fe-453a-ae2b-7dec205c7931","Type":"ContainerStarted","Data":"d051af7d6508c695cb28214fe3b075796fbd5aa8fdbc7b2099cfc5317ed1721f"} Oct 30 19:54:11.681718 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:11.681686 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:12.164789 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.164740 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" podUID="847ac050-31eb-4adb-9457-2daf3e4ae09b" containerName="main" containerID="cri-o://4d51faca20b3e3bd6a5333b80ed6653439c5145ac9b10be150e3c771d18514e9" gracePeriod=45 Oct 30 19:54:12.324509 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.324477 5047 generic.go:356] "Generic (PLEG): container finished" podID="847ac050-31eb-4adb-9457-2daf3e4ae09b" containerID="4d51faca20b3e3bd6a5333b80ed6653439c5145ac9b10be150e3c771d18514e9" exitCode=0 Oct 30 19:54:12.324787 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.324551 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" event={"ID":"847ac050-31eb-4adb-9457-2daf3e4ae09b","Type":"ContainerDied","Data":"4d51faca20b3e3bd6a5333b80ed6653439c5145ac9b10be150e3c771d18514e9"} Oct 30 19:54:12.334268 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.334227 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" podUID="847ac050-31eb-4adb-9457-2daf3e4ae09b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": dial tcp 10.132.0.190:8000: connect: connection refused" Oct 30 19:54:12.554308 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.554203 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:54:12.632034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.630849 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" podStartSLOduration=2.630828557 podStartE2EDuration="2.630828557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:11.377775339 +0000 UTC m=+3004.452230711" watchObservedRunningTime="2026-10-30 19:54:12.630828557 +0000 UTC m=+3005.705283916" Oct 30 19:54:12.690666 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.690579 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:12.715453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.715410 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847ac050-31eb-4adb-9457-2daf3e4ae09b-kube-api-access-gqcb2\" (UniqueName: \"kubernetes.io/projected/847ac050-31eb-4adb-9457-2daf3e4ae09b-kube-api-access-gqcb2\") pod \"847ac050-31eb-4adb-9457-2daf3e4ae09b\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " Oct 30 19:54:12.715658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.715500 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-dshm\") pod \"847ac050-31eb-4adb-9457-2daf3e4ae09b\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " Oct 30 19:54:12.715658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.715567 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-home\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-home\") pod \"847ac050-31eb-4adb-9457-2daf3e4ae09b\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " Oct 30 19:54:12.715830 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.715808 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-model-cache\") pod \"847ac050-31eb-4adb-9457-2daf3e4ae09b\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " Oct 30 19:54:12.715939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.715918 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/847ac050-31eb-4adb-9457-2daf3e4ae09b-tls-certs\" (UniqueName: \"kubernetes.io/secret/847ac050-31eb-4adb-9457-2daf3e4ae09b-tls-certs\") pod \"847ac050-31eb-4adb-9457-2daf3e4ae09b\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " Oct 30 19:54:12.716052 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.716000 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-tmp-dir\") pod \"847ac050-31eb-4adb-9457-2daf3e4ae09b\" (UID: \"847ac050-31eb-4adb-9457-2daf3e4ae09b\") " Oct 30 19:54:12.716122 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.716053 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-home" pod "847ac050-31eb-4adb-9457-2daf3e4ae09b" (UID: "847ac050-31eb-4adb-9457-2daf3e4ae09b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:12.716189 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.716132 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-model-cache" pod "847ac050-31eb-4adb-9457-2daf3e4ae09b" (UID: "847ac050-31eb-4adb-9457-2daf3e4ae09b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:12.716396 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.716372 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-tmp-dir" pod "847ac050-31eb-4adb-9457-2daf3e4ae09b" (UID: "847ac050-31eb-4adb-9457-2daf3e4ae09b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:12.716804 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.716774 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:54:12.716929 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.716807 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:54:12.716929 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.716821 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:54:12.719131 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.719104 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847ac050-31eb-4adb-9457-2daf3e4ae09b-kube-api-access-gqcb2" pod "847ac050-31eb-4adb-9457-2daf3e4ae09b" (UID: "847ac050-31eb-4adb-9457-2daf3e4ae09b"). InnerVolumeSpecName "kube-api-access-gqcb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:12.720518 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.720492 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847ac050-31eb-4adb-9457-2daf3e4ae09b-tls-certs" pod "847ac050-31eb-4adb-9457-2daf3e4ae09b" (UID: "847ac050-31eb-4adb-9457-2daf3e4ae09b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:12.720674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.720636 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-dshm" pod "847ac050-31eb-4adb-9457-2daf3e4ae09b" (UID: "847ac050-31eb-4adb-9457-2daf3e4ae09b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:12.818313 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.818278 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847ac050-31eb-4adb-9457-2daf3e4ae09b-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:54:12.818313 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.818315 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqcb2\" (UniqueName: \"kubernetes.io/projected/847ac050-31eb-4adb-9457-2daf3e4ae09b-kube-api-access-gqcb2\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:54:12.818537 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:12.818330 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847ac050-31eb-4adb-9457-2daf3e4ae09b-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:54:13.335491 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:13.335460 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" Oct 30 19:54:13.335925 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:13.335474 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q" event={"ID":"847ac050-31eb-4adb-9457-2daf3e4ae09b","Type":"ContainerDied","Data":"3b667a59e667547a43be9411719b7aa3ce003656ab8c1b2f6b32c80f76ff8968"} Oct 30 19:54:13.336125 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:13.335960 5047 scope.go:122] "RemoveContainer" containerID="4d51faca20b3e3bd6a5333b80ed6653439c5145ac9b10be150e3c771d18514e9" Oct 30 19:54:13.336489 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:13.336458 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:13.338822 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:13.338801 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:54:13.384341 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:13.384303 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q"] Oct 30 19:54:13.390317 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:13.390285 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cd4bde/auth-test-model-cd4bde-kserve-5bc7456557-9qs4q"] Oct 30 19:54:15.128765 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:15.128365 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="847ac050-31eb-4adb-9457-2daf3e4ae09b" path="/var/lib/kubelet/pods/847ac050-31eb-4adb-9457-2daf3e4ae09b/volumes" Oct 30 19:54:16.238673 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.237291 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:16.276829 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.259225 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" podUID="37636069-645a-4941-97fd-a2fa4b3df247" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:16.377028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.376319 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:16.409172 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.391323 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z"] Oct 30 19:54:16.409172 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.393306 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:16.409172 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.408327 5047 cpu_manager.go:414] "containerMap: removing container" podUID="847ac050-31eb-4adb-9457-2daf3e4ae09b" containerName="main" Oct 30 19:54:16.409172 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.408354 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847ac050-31eb-4adb-9457-2daf3e4ae09b" containerName="main" Oct 30 19:54:16.415204 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.410099 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="847ac050-31eb-4adb-9457-2daf3e4ae09b" containerName="main" Oct 30 19:54:16.439112 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.431682 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z"] Oct 30 19:54:16.439112 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.431925 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" Oct 30 19:54:16.465030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.459490 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ed2ede-dockercfg-rwfdf\"" Oct 30 19:54:16.465030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.459682 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ed2ede\"" Oct 30 19:54:16.722351 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.722314 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrlfm\" (UniqueName: \"kubernetes.io/projected/1258562d-1c87-43fe-b887-42591c2460d6-kube-api-access-rrlfm\") pod \"payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z\" (UID: \"1258562d-1c87-43fe-b887-42591c2460d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" Oct 30 19:54:16.722590 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.722566 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1258562d-1c87-43fe-b887-42591c2460d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z\" (UID: \"1258562d-1c87-43fe-b887-42591c2460d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" Oct 30 19:54:16.824272 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.824180 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1258562d-1c87-43fe-b887-42591c2460d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z\" (UID: \"1258562d-1c87-43fe-b887-42591c2460d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" Oct 30 19:54:16.824438 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.824392 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrlfm\" (UniqueName: \"kubernetes.io/projected/1258562d-1c87-43fe-b887-42591c2460d6-kube-api-access-rrlfm\") pod \"payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z\" (UID: \"1258562d-1c87-43fe-b887-42591c2460d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" Oct 30 19:54:16.825159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.825134 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1258562d-1c87-43fe-b887-42591c2460d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z\" (UID: \"1258562d-1c87-43fe-b887-42591c2460d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" Oct 30 19:54:16.865635 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.865602 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrlfm\" (UniqueName: \"kubernetes.io/projected/1258562d-1c87-43fe-b887-42591c2460d6-kube-api-access-rrlfm\") pod \"payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z\" (UID: \"1258562d-1c87-43fe-b887-42591c2460d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" Oct 30 19:54:16.919174 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:16.917286 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" Oct 30 19:54:17.622466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:17.622436 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z"] Oct 30 19:54:17.623573 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:54:17.623542 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1258562d_1c87_43fe_b887_42591c2460d6.slice/crio-6b15976360894e6cde1624dc69444601cc7e3954cf5f1797d074d3fc65e665c7 WatchSource:0}: Error finding container 6b15976360894e6cde1624dc69444601cc7e3954cf5f1797d074d3fc65e665c7: Status 404 returned error can't find the container with id 6b15976360894e6cde1624dc69444601cc7e3954cf5f1797d074d3fc65e665c7 Oct 30 19:54:17.706811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:17.706777 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" event={"ID":"1258562d-1c87-43fe-b887-42591c2460d6","Type":"ContainerStarted","Data":"6b15976360894e6cde1624dc69444601cc7e3954cf5f1797d074d3fc65e665c7"} Oct 30 19:54:18.716119 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:18.716081 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" event={"ID":"1258562d-1c87-43fe-b887-42591c2460d6","Type":"ContainerStarted","Data":"83f0de5c328693acc831ebcd851bb8d7ec5c9d6d124078a32c1b7c47aac14d31"} Oct 30 19:54:18.716523 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:18.716157 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" Oct 30 19:54:18.765031 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:18.764909 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" podStartSLOduration=3.764884582 podStartE2EDuration="3.764884582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:18.754372257 +0000 UTC m=+3011.828827618" watchObservedRunningTime="2026-10-30 19:54:18.764884582 +0000 UTC m=+3011.839339944" Oct 30 19:54:24.489196 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:24.489147 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:54:24.489630 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:24.489239 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" podUID="2547bf26-2d3b-4565-b16c-c37d17d410d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:25.843564 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:25.843529 5047 generic.go:356] "Generic (PLEG): container finished" podID="2547bf26-2d3b-4565-b16c-c37d17d410d7" containerID="b55a051a5b0d7c5e39a75bf8e4f9b5812b6845acf9cc2d890ed4a9cc885b843e" exitCode=0 Oct 30 19:54:25.844046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:25.843602 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" event={"ID":"2547bf26-2d3b-4565-b16c-c37d17d410d7","Type":"ContainerDied","Data":"b55a051a5b0d7c5e39a75bf8e4f9b5812b6845acf9cc2d890ed4a9cc885b843e"} Oct 30 19:54:25.844193 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:25.844174 5047 scope.go:122] "RemoveContainer" containerID="b55a051a5b0d7c5e39a75bf8e4f9b5812b6845acf9cc2d890ed4a9cc885b843e" Oct 30 19:54:25.846175 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:25.846148 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:25.846240 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:25.846222 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:25.846277 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:25.846255 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:26.867879 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:26.867840 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" event={"ID":"2547bf26-2d3b-4565-b16c-c37d17d410d7","Type":"ContainerStarted","Data":"44b7bcfcb36cb8745819e8aed0eb637513b29a0ca49dbf2ca71ce385a7679e31"} Oct 30 19:54:27.852027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:27.851414 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:54:27.852027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:27.851545 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 19:54:27.852027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:27.851617 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" podUID="2547bf26-2d3b-4565-b16c-c37d17d410d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 19:54:28.851457 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:28.851414 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 19:54:28.853176 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:28.853142 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" podUID="2547bf26-2d3b-4565-b16c-c37d17d410d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 19:54:29.727147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:29.727116 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" Oct 30 19:54:29.854839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:29.853418 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 19:54:29.854839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:29.853484 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" podUID="2547bf26-2d3b-4565-b16c-c37d17d410d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 19:54:30.851039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:30.845359 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 19:54:30.851039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:30.845424 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" podUID="2547bf26-2d3b-4565-b16c-c37d17d410d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 19:54:31.852125 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:31.852086 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 19:54:31.852748 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:31.852712 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" podUID="2547bf26-2d3b-4565-b16c-c37d17d410d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 19:54:32.849521 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:32.849486 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:54:32.849915 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:32.849888 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:54:32.850675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:32.850656 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:54:39.602616 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:54:39.602579 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:54:40.087126 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:40.087085 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt"] Oct 30 19:54:40.087919 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:40.087853 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" podUID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" containerName="istio-proxy" containerID="cri-o://ac9500c921bd6d33247a6bd3a311e30b30c0245b4f25ab9084e0e842ead83283" gracePeriod=30 Oct 30 19:54:41.357410 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:41.357377 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf"] Oct 30 19:54:43.561728 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:43.561687 5047 patch_prober.go:29] interesting pod/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:54:43.562180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:43.561764 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" podUID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:54:45.393305 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.393238 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:54:45.457123 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.457082 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-podinfo\") pod \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " Oct 30 19:54:45.457313 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.457195 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b67c1b-d500-4e75-affb-e748e37b0cfb-kube-api-access-b6lm5\" (UniqueName: \"kubernetes.io/projected/f5b67c1b-d500-4e75-affb-e748e37b0cfb-kube-api-access-b6lm5\") pod \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " Oct 30 19:54:45.457313 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.457240 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-workload-socket\") pod \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " Oct 30 19:54:45.457313 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.457271 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-envoy\") pod \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " Oct 30 19:54:45.457313 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.457305 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-credential-socket\") pod \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " Oct 30 19:54:45.457561 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.457466 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-workload-certs\") pod \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " Oct 30 19:54:45.457561 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.457512 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-data\") pod \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " Oct 30 19:54:45.457677 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.457655 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istiod-ca-cert\") pod \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " Oct 30 19:54:45.458568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.457773 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-token\" (UniqueName: \"kubernetes.io/projected/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-token\") pod \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\" (UID: \"f5b67c1b-d500-4e75-affb-e748e37b0cfb\") " Oct 30 19:54:45.458568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.458247 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-data" pod "f5b67c1b-d500-4e75-affb-e748e37b0cfb" (UID: "f5b67c1b-d500-4e75-affb-e748e37b0cfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:45.458568 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.458500 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istiod-ca-cert" pod "f5b67c1b-d500-4e75-affb-e748e37b0cfb" (UID: "f5b67c1b-d500-4e75-affb-e748e37b0cfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:45.458892 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.458607 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:54:45.458892 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.458759 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-workload-socket" pod "f5b67c1b-d500-4e75-affb-e748e37b0cfb" (UID: "f5b67c1b-d500-4e75-affb-e748e37b0cfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:45.458892 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.458816 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-credential-socket" pod "f5b67c1b-d500-4e75-affb-e748e37b0cfb" (UID: "f5b67c1b-d500-4e75-affb-e748e37b0cfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:45.458892 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.458866 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-workload-certs" pod "f5b67c1b-d500-4e75-affb-e748e37b0cfb" (UID: "f5b67c1b-d500-4e75-affb-e748e37b0cfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:45.460916 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.460886 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-podinfo" pod "f5b67c1b-d500-4e75-affb-e748e37b0cfb" (UID: "f5b67c1b-d500-4e75-affb-e748e37b0cfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:54:45.461395 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.461363 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-envoy" pod "f5b67c1b-d500-4e75-affb-e748e37b0cfb" (UID: "f5b67c1b-d500-4e75-affb-e748e37b0cfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:45.461517 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.461492 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-token" pod "f5b67c1b-d500-4e75-affb-e748e37b0cfb" (UID: "f5b67c1b-d500-4e75-affb-e748e37b0cfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:45.461662 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.461638 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b67c1b-d500-4e75-affb-e748e37b0cfb-kube-api-access-b6lm5" pod "f5b67c1b-d500-4e75-affb-e748e37b0cfb" (UID: "f5b67c1b-d500-4e75-affb-e748e37b0cfb"). InnerVolumeSpecName "kube-api-access-b6lm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:45.560564 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.560525 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:54:45.560564 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.560558 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:54:45.560564 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.560569 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:54:45.560818 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.560591 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6lm5\" (UniqueName: \"kubernetes.io/projected/f5b67c1b-d500-4e75-affb-e748e37b0cfb-kube-api-access-b6lm5\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:54:45.560818 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.560601 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:54:45.560818 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.560610 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:54:45.560818 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.560618 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:54:45.560818 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:45.560627 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b67c1b-d500-4e75-affb-e748e37b0cfb-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:54:46.143702 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.143670 5047 generic.go:356] "Generic (PLEG): container finished" podID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" containerID="ac9500c921bd6d33247a6bd3a311e30b30c0245b4f25ab9084e0e842ead83283" exitCode=0 Oct 30 19:54:46.143908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.143723 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" Oct 30 19:54:46.143908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.143752 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" event={"ID":"f5b67c1b-d500-4e75-affb-e748e37b0cfb","Type":"ContainerDied","Data":"ac9500c921bd6d33247a6bd3a311e30b30c0245b4f25ab9084e0e842ead83283"} Oct 30 19:54:46.143908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.143809 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt" event={"ID":"f5b67c1b-d500-4e75-affb-e748e37b0cfb","Type":"ContainerDied","Data":"b232a20495b70c172db8c96e3cc0fbf51c8fc2da3c5b0ec80c58c079c99ddb9e"} Oct 30 19:54:46.143908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.143835 5047 scope.go:122] "RemoveContainer" containerID="ac9500c921bd6d33247a6bd3a311e30b30c0245b4f25ab9084e0e842ead83283" Oct 30 19:54:46.157565 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.157536 5047 scope.go:122] "RemoveContainer" containerID="d3033c99f6d6304cf6d9b723014b787b4375f0c84c2a148a5945cee80f16fe82" Oct 30 19:54:46.167910 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.167890 5047 scope.go:122] "RemoveContainer" containerID="ac9500c921bd6d33247a6bd3a311e30b30c0245b4f25ab9084e0e842ead83283" Oct 30 19:54:46.168288 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:54:46.168265 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9500c921bd6d33247a6bd3a311e30b30c0245b4f25ab9084e0e842ead83283\": container with ID starting with ac9500c921bd6d33247a6bd3a311e30b30c0245b4f25ab9084e0e842ead83283 not found: ID does not exist" containerID="ac9500c921bd6d33247a6bd3a311e30b30c0245b4f25ab9084e0e842ead83283" Oct 30 19:54:46.168392 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.168297 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9500c921bd6d33247a6bd3a311e30b30c0245b4f25ab9084e0e842ead83283"} err="failed to get container status \"ac9500c921bd6d33247a6bd3a311e30b30c0245b4f25ab9084e0e842ead83283\": rpc error: code = NotFound desc = could not find container \"ac9500c921bd6d33247a6bd3a311e30b30c0245b4f25ab9084e0e842ead83283\": container with ID starting with ac9500c921bd6d33247a6bd3a311e30b30c0245b4f25ab9084e0e842ead83283 not found: ID does not exist" Oct 30 19:54:46.168392 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.168316 5047 scope.go:122] "RemoveContainer" containerID="d3033c99f6d6304cf6d9b723014b787b4375f0c84c2a148a5945cee80f16fe82" Oct 30 19:54:46.168629 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:54:46.168607 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3033c99f6d6304cf6d9b723014b787b4375f0c84c2a148a5945cee80f16fe82\": container with ID starting with d3033c99f6d6304cf6d9b723014b787b4375f0c84c2a148a5945cee80f16fe82 not found: ID does not exist" containerID="d3033c99f6d6304cf6d9b723014b787b4375f0c84c2a148a5945cee80f16fe82" Oct 30 19:54:46.168690 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.168637 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3033c99f6d6304cf6d9b723014b787b4375f0c84c2a148a5945cee80f16fe82"} err="failed to get container status \"d3033c99f6d6304cf6d9b723014b787b4375f0c84c2a148a5945cee80f16fe82\": rpc error: code = NotFound desc = could not find container \"d3033c99f6d6304cf6d9b723014b787b4375f0c84c2a148a5945cee80f16fe82\": container with ID starting with d3033c99f6d6304cf6d9b723014b787b4375f0c84c2a148a5945cee80f16fe82 not found: ID does not exist" Oct 30 19:54:46.201674 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.201640 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt"] Oct 30 19:54:46.205425 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.205398 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-852709-openshift-default-557fdb8995-26hnt"] Oct 30 19:54:46.358404 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.358364 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" podUID="5744eb2c-c3d8-4ad7-8619-21075c182483" containerName="payload-processing" containerID="cri-o://7985e863553b000438abf7abd280a4622afb40f77b99fb25b5d715bce78fdd2b" gracePeriod=25 Oct 30 19:54:46.693658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.693634 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" Oct 30 19:54:46.777846 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.777752 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5744eb2c-c3d8-4ad7-8619-21075c182483-kube-api-access-2zg5q\" (UniqueName: \"kubernetes.io/projected/5744eb2c-c3d8-4ad7-8619-21075c182483-kube-api-access-2zg5q\") pod \"5744eb2c-c3d8-4ad7-8619-21075c182483\" (UID: \"5744eb2c-c3d8-4ad7-8619-21075c182483\") " Oct 30 19:54:46.777846 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.777836 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5744eb2c-c3d8-4ad7-8619-21075c182483-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5744eb2c-c3d8-4ad7-8619-21075c182483-plugins-config-volume\") pod \"5744eb2c-c3d8-4ad7-8619-21075c182483\" (UID: \"5744eb2c-c3d8-4ad7-8619-21075c182483\") " Oct 30 19:54:46.778314 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.778279 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5744eb2c-c3d8-4ad7-8619-21075c182483-plugins-config-volume" pod "5744eb2c-c3d8-4ad7-8619-21075c182483" (UID: "5744eb2c-c3d8-4ad7-8619-21075c182483"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:46.778688 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.778669 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5744eb2c-c3d8-4ad7-8619-21075c182483-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:54:46.780735 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.780700 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5744eb2c-c3d8-4ad7-8619-21075c182483-kube-api-access-2zg5q" pod "5744eb2c-c3d8-4ad7-8619-21075c182483" (UID: "5744eb2c-c3d8-4ad7-8619-21075c182483"). InnerVolumeSpecName "kube-api-access-2zg5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:46.880127 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:46.880089 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zg5q\" (UniqueName: \"kubernetes.io/projected/5744eb2c-c3d8-4ad7-8619-21075c182483-kube-api-access-2zg5q\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:54:47.053661 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.053575 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" path="/var/lib/kubelet/pods/f5b67c1b-d500-4e75-affb-e748e37b0cfb/volumes" Oct 30 19:54:47.153583 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.153548 5047 generic.go:356] "Generic (PLEG): container finished" podID="5744eb2c-c3d8-4ad7-8619-21075c182483" containerID="7985e863553b000438abf7abd280a4622afb40f77b99fb25b5d715bce78fdd2b" exitCode=0 Oct 30 19:54:47.153753 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.153596 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" Oct 30 19:54:47.153753 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.153634 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" event={"ID":"5744eb2c-c3d8-4ad7-8619-21075c182483","Type":"ContainerDied","Data":"7985e863553b000438abf7abd280a4622afb40f77b99fb25b5d715bce78fdd2b"} Oct 30 19:54:47.153753 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.153694 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf" event={"ID":"5744eb2c-c3d8-4ad7-8619-21075c182483","Type":"ContainerDied","Data":"b25fa6855efe2733769909e3f4e226f3557bd50389bccd7c8be0801a574b8da9"} Oct 30 19:54:47.153753 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.153724 5047 scope.go:122] "RemoveContainer" containerID="7985e863553b000438abf7abd280a4622afb40f77b99fb25b5d715bce78fdd2b" Oct 30 19:54:47.165610 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.165583 5047 scope.go:122] "RemoveContainer" containerID="7985e863553b000438abf7abd280a4622afb40f77b99fb25b5d715bce78fdd2b" Oct 30 19:54:47.165882 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:54:47.165861 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7985e863553b000438abf7abd280a4622afb40f77b99fb25b5d715bce78fdd2b\": container with ID starting with 7985e863553b000438abf7abd280a4622afb40f77b99fb25b5d715bce78fdd2b not found: ID does not exist" containerID="7985e863553b000438abf7abd280a4622afb40f77b99fb25b5d715bce78fdd2b" Oct 30 19:54:47.165939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.165892 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7985e863553b000438abf7abd280a4622afb40f77b99fb25b5d715bce78fdd2b"} err="failed to get container status \"7985e863553b000438abf7abd280a4622afb40f77b99fb25b5d715bce78fdd2b\": rpc error: code = NotFound desc = could not find container \"7985e863553b000438abf7abd280a4622afb40f77b99fb25b5d715bce78fdd2b\": container with ID starting with 7985e863553b000438abf7abd280a4622afb40f77b99fb25b5d715bce78fdd2b not found: ID does not exist" Oct 30 19:54:47.205193 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.205160 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf"] Oct 30 19:54:47.209153 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.209127 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-5dfd4c854b-m9qvf"] Oct 30 19:54:47.951115 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.951083 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7"] Oct 30 19:54:47.951867 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.951852 5047 cpu_manager.go:414] "containerMap: removing container" podUID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" containerName="istio-proxy" Oct 30 19:54:47.951910 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.951868 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" containerName="istio-proxy" Oct 30 19:54:47.951910 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.951903 5047 cpu_manager.go:414] "containerMap: removing container" podUID="5744eb2c-c3d8-4ad7-8619-21075c182483" containerName="payload-processing" Oct 30 19:54:47.951910 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.951908 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5744eb2c-c3d8-4ad7-8619-21075c182483" containerName="payload-processing" Oct 30 19:54:47.952068 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.951921 5047 cpu_manager.go:414] "containerMap: removing container" podUID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" containerName="istio-proxy" Oct 30 19:54:47.952068 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.951926 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" containerName="istio-proxy" Oct 30 19:54:47.952068 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.952055 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" containerName="istio-proxy" Oct 30 19:54:47.952166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.952072 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="5744eb2c-c3d8-4ad7-8619-21075c182483" containerName="payload-processing" Oct 30 19:54:47.952166 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.952078 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b67c1b-d500-4e75-affb-e748e37b0cfb" containerName="istio-proxy" Oct 30 19:54:47.957394 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.957377 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" Oct 30 19:54:47.970037 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.969975 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-59f9ed\"" Oct 30 19:54:47.974152 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:47.974125 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7"] Oct 30 19:54:48.046591 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.046549 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-59f9ed-dockercfg-dkmkh\"" Oct 30 19:54:48.114053 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.085076 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r"] Oct 30 19:54:48.154303 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.154260 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7gpm\" (UniqueName: \"kubernetes.io/projected/d82abce5-4bc3-4a06-aa68-4e688cd1702e-kube-api-access-t7gpm\") pod \"payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7\" (UID: \"d82abce5-4bc3-4a06-aa68-4e688cd1702e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" Oct 30 19:54:48.157428 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.157395 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d82abce5-4bc3-4a06-aa68-4e688cd1702e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7\" (UID: \"d82abce5-4bc3-4a06-aa68-4e688cd1702e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" Oct 30 19:54:48.192916 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.191260 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" Oct 30 19:54:48.210645 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.210465 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r"] Oct 30 19:54:48.290800 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.290757 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7gpm\" (UniqueName: \"kubernetes.io/projected/d82abce5-4bc3-4a06-aa68-4e688cd1702e-kube-api-access-t7gpm\") pod \"payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7\" (UID: \"d82abce5-4bc3-4a06-aa68-4e688cd1702e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" Oct 30 19:54:48.291187 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.291158 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d82abce5-4bc3-4a06-aa68-4e688cd1702e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7\" (UID: \"d82abce5-4bc3-4a06-aa68-4e688cd1702e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" Oct 30 19:54:48.291449 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.291426 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14168ba7-e407-457a-8a39-b4d0c266135a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r\" (UID: \"14168ba7-e407-457a-8a39-b4d0c266135a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" Oct 30 19:54:48.291807 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.291787 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dgtp\" (UniqueName: \"kubernetes.io/projected/14168ba7-e407-457a-8a39-b4d0c266135a-kube-api-access-2dgtp\") pod \"payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r\" (UID: \"14168ba7-e407-457a-8a39-b4d0c266135a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" Oct 30 19:54:48.294058 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.294037 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d82abce5-4bc3-4a06-aa68-4e688cd1702e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7\" (UID: \"d82abce5-4bc3-4a06-aa68-4e688cd1702e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" Oct 30 19:54:48.313216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.313184 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7gpm\" (UniqueName: \"kubernetes.io/projected/d82abce5-4bc3-4a06-aa68-4e688cd1702e-kube-api-access-t7gpm\") pod \"payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7\" (UID: \"d82abce5-4bc3-4a06-aa68-4e688cd1702e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" Oct 30 19:54:48.393629 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.393595 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14168ba7-e407-457a-8a39-b4d0c266135a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r\" (UID: \"14168ba7-e407-457a-8a39-b4d0c266135a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" Oct 30 19:54:48.393804 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.393728 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dgtp\" (UniqueName: \"kubernetes.io/projected/14168ba7-e407-457a-8a39-b4d0c266135a-kube-api-access-2dgtp\") pod \"payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r\" (UID: \"14168ba7-e407-457a-8a39-b4d0c266135a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" Oct 30 19:54:48.394429 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.394407 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14168ba7-e407-457a-8a39-b4d0c266135a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r\" (UID: \"14168ba7-e407-457a-8a39-b4d0c266135a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" Oct 30 19:54:48.411363 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.411333 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dgtp\" (UniqueName: \"kubernetes.io/projected/14168ba7-e407-457a-8a39-b4d0c266135a-kube-api-access-2dgtp\") pod \"payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r\" (UID: \"14168ba7-e407-457a-8a39-b4d0c266135a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" Oct 30 19:54:48.573624 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.573536 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" Oct 30 19:54:48.610527 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.610485 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" Oct 30 19:54:48.899081 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.898670 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7"] Oct 30 19:54:48.900078 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:54:48.900040 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82abce5_4bc3_4a06_aa68_4e688cd1702e.slice/crio-5d44a84d35d69fdd743bfc76b6c197f9fbdba948f68990e7f80c7c3a8264847a WatchSource:0}: Error finding container 5d44a84d35d69fdd743bfc76b6c197f9fbdba948f68990e7f80c7c3a8264847a: Status 404 returned error can't find the container with id 5d44a84d35d69fdd743bfc76b6c197f9fbdba948f68990e7f80c7c3a8264847a Oct 30 19:54:48.936618 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:48.936590 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r"] Oct 30 19:54:48.944566 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:54:48.944539 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14168ba7_e407_457a_8a39_b4d0c266135a.slice/crio-e8123773d997b8fba0ab210d9c15cfb35a6471164d099843db43335f7d7b5ce9 WatchSource:0}: Error finding container e8123773d997b8fba0ab210d9c15cfb35a6471164d099843db43335f7d7b5ce9: Status 404 returned error can't find the container with id e8123773d997b8fba0ab210d9c15cfb35a6471164d099843db43335f7d7b5ce9 Oct 30 19:54:49.053691 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:49.053656 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5744eb2c-c3d8-4ad7-8619-21075c182483" path="/var/lib/kubelet/pods/5744eb2c-c3d8-4ad7-8619-21075c182483/volumes" Oct 30 19:54:49.299973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:49.299862 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" event={"ID":"d82abce5-4bc3-4a06-aa68-4e688cd1702e","Type":"ContainerStarted","Data":"1e2342f5bd2b75c6c6c252ebeda84410590514fd13b61e775db8e3ee725e23dc"} Oct 30 19:54:49.299973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:49.299934 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" Oct 30 19:54:49.299973 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:49.299957 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" event={"ID":"d82abce5-4bc3-4a06-aa68-4e688cd1702e","Type":"ContainerStarted","Data":"5d44a84d35d69fdd743bfc76b6c197f9fbdba948f68990e7f80c7c3a8264847a"} Oct 30 19:54:49.301734 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:49.301698 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" event={"ID":"14168ba7-e407-457a-8a39-b4d0c266135a","Type":"ContainerStarted","Data":"4c608ef38e9e55b2ca7e5fcf3d4130b886effe03b01af298b68936f86a6d15dc"} Oct 30 19:54:49.301886 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:49.301761 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" event={"ID":"14168ba7-e407-457a-8a39-b4d0c266135a","Type":"ContainerStarted","Data":"e8123773d997b8fba0ab210d9c15cfb35a6471164d099843db43335f7d7b5ce9"} Oct 30 19:54:49.301886 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:49.301793 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" Oct 30 19:54:49.337645 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:49.337597 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" podStartSLOduration=2.337583286 podStartE2EDuration="2.337583286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:49.33431706 +0000 UTC m=+3042.408772420" watchObservedRunningTime="2026-10-30 19:54:49.337583286 +0000 UTC m=+3042.412038644" Oct 30 19:54:49.369653 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:49.369605 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" podStartSLOduration=1.3695909290000001 podStartE2EDuration="1.369590929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:49.368660424 +0000 UTC m=+3042.443115783" watchObservedRunningTime="2026-10-30 19:54:49.369590929 +0000 UTC m=+3042.444046337" Oct 30 19:54:53.154813 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.154776 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj"] Oct 30 19:54:53.160688 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.160667 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.184412 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.184383 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-cd4bde-openshift-default-dockercfg-dhs49\"" Oct 30 19:54:53.188860 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.188829 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj"] Oct 30 19:54:53.201359 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.201327 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-workload-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.201502 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.201464 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istiod-ca-cert\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.201566 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.201543 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5hbk\" (UniqueName: \"kubernetes.io/projected/c0cd76c6-9b67-4719-9d13-82315cd9f77a-kube-api-access-j5hbk\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.201616 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.201583 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-token\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.201666 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.201625 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-envoy\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.201666 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.201646 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-credential-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.201743 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.201727 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-data\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.201806 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.201789 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-workload-certs\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.201933 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.201908 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-podinfo\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.303422 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.303389 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-workload-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.303618 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.303480 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istiod-ca-cert\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.303618 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.303514 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5hbk\" (UniqueName: \"kubernetes.io/projected/c0cd76c6-9b67-4719-9d13-82315cd9f77a-kube-api-access-j5hbk\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.303618 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.303553 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-token\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.303618 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.303613 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-envoy\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.303870 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.303643 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-credential-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.303870 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.303675 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-data\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.303870 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.303723 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-workload-certs\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.304052 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.303878 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-podinfo\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.304052 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.303882 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-workload-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.304052 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.304034 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-credential-socket\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.304220 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.304143 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-data\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.304220 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.304197 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-workload-certs\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.304563 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.304536 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istiod-ca-cert\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.305915 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.305892 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-envoy\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.306324 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.306308 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-podinfo\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.320777 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.320753 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5hbk\" (UniqueName: \"kubernetes.io/projected/c0cd76c6-9b67-4719-9d13-82315cd9f77a-kube-api-access-j5hbk\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.320877 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.320833 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-token\") pod \"e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.477423 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.477340 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:53.798839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.798807 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj"] Oct 30 19:54:53.799068 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:54:53.799044 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cd76c6_9b67_4719_9d13_82315cd9f77a.slice/crio-fcd3d12656d612173de2fec06f288fb140f304511b44eb828247b9805e1c92ed WatchSource:0}: Error finding container fcd3d12656d612173de2fec06f288fb140f304511b44eb828247b9805e1c92ed: Status 404 returned error can't find the container with id fcd3d12656d612173de2fec06f288fb140f304511b44eb828247b9805e1c92ed Oct 30 19:54:53.802274 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.802242 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:53.802383 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.802307 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:53.802383 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:53.802338 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:54.346409 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:54.346368 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" event={"ID":"c0cd76c6-9b67-4719-9d13-82315cd9f77a","Type":"ContainerStarted","Data":"869409a721cfd2f3b4a960818549deec0a34d3edcbcb9adddc1abe3e761fd5c1"} Oct 30 19:54:54.346845 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:54.346435 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" event={"ID":"c0cd76c6-9b67-4719-9d13-82315cd9f77a","Type":"ContainerStarted","Data":"fcd3d12656d612173de2fec06f288fb140f304511b44eb828247b9805e1c92ed"} Oct 30 19:54:54.408271 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:54.408216 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" podStartSLOduration=1.408202255 podStartE2EDuration="1.408202255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:54.394481771 +0000 UTC m=+3047.468937131" watchObservedRunningTime="2026-10-30 19:54:54.408202255 +0000 UTC m=+3047.482657611" Oct 30 19:54:54.478430 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:54.478398 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:54:54.479778 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:54.479749 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 19:54:54.479904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:54.479813 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" podUID="c0cd76c6-9b67-4719-9d13-82315cd9f77a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 19:54:55.478649 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:55.478612 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 19:54:55.479082 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:55.478676 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" podUID="c0cd76c6-9b67-4719-9d13-82315cd9f77a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 19:54:56.478624 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:56.478579 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 19:54:56.478847 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:56.478652 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" podUID="c0cd76c6-9b67-4719-9d13-82315cd9f77a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 19:54:57.477816 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:57.477775 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 19:54:57.478132 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:57.477840 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" podUID="c0cd76c6-9b67-4719-9d13-82315cd9f77a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 19:54:58.478596 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:58.478560 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 19:54:58.479201 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:58.479146 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" podUID="c0cd76c6-9b67-4719-9d13-82315cd9f77a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 19:54:59.165158 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.165100 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj"] Oct 30 19:54:59.177152 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.177120 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj"] Oct 30 19:54:59.177340 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.177251 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 19:54:59.198644 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.198607 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bf314f-serving-cert\"" Oct 30 19:54:59.271577 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.267626 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62xrj\" (UniqueName: \"kubernetes.io/projected/70395564-6b94-4b5c-af03-06efe90cb46a-kube-api-access-62xrj\") pod \"maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj\" (UID: \"70395564-6b94-4b5c-af03-06efe90cb46a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 19:54:59.271577 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.267719 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70395564-6b94-4b5c-af03-06efe90cb46a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj\" (UID: \"70395564-6b94-4b5c-af03-06efe90cb46a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 19:54:59.271577 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.267930 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70395564-6b94-4b5c-af03-06efe90cb46a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj\" (UID: \"70395564-6b94-4b5c-af03-06efe90cb46a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 19:54:59.271577 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.267994 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70395564-6b94-4b5c-af03-06efe90cb46a-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj\" (UID: \"70395564-6b94-4b5c-af03-06efe90cb46a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 19:54:59.404201 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.389787 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70395564-6b94-4b5c-af03-06efe90cb46a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj\" (UID: \"70395564-6b94-4b5c-af03-06efe90cb46a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 19:54:59.404735 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.404693 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70395564-6b94-4b5c-af03-06efe90cb46a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj\" (UID: \"70395564-6b94-4b5c-af03-06efe90cb46a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 19:54:59.410028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.404943 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70395564-6b94-4b5c-af03-06efe90cb46a-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj\" (UID: \"70395564-6b94-4b5c-af03-06efe90cb46a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 19:54:59.419195 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.410623 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62xrj\" (UniqueName: \"kubernetes.io/projected/70395564-6b94-4b5c-af03-06efe90cb46a-kube-api-access-62xrj\") pod \"maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj\" (UID: \"70395564-6b94-4b5c-af03-06efe90cb46a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 19:54:59.424968 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.424179 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70395564-6b94-4b5c-af03-06efe90cb46a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj\" (UID: \"70395564-6b94-4b5c-af03-06efe90cb46a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 19:54:59.425743 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.425716 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70395564-6b94-4b5c-af03-06efe90cb46a-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj\" (UID: \"70395564-6b94-4b5c-af03-06efe90cb46a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 19:54:59.426849 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.426821 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70395564-6b94-4b5c-af03-06efe90cb46a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj\" (UID: \"70395564-6b94-4b5c-af03-06efe90cb46a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 19:54:59.467619 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.459740 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62xrj\" (UniqueName: \"kubernetes.io/projected/70395564-6b94-4b5c-af03-06efe90cb46a-kube-api-access-62xrj\") pod \"maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj\" (UID: \"70395564-6b94-4b5c-af03-06efe90cb46a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 19:54:59.478643 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.478606 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 19:54:59.479083 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.478676 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" podUID="c0cd76c6-9b67-4719-9d13-82315cd9f77a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 19:54:59.507306 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.507265 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 19:54:59.861911 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:54:59.861871 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70395564_6b94_4b5c_af03_06efe90cb46a.slice/crio-3eca61f088f9ff743f2af52e51dbd367624b5fa4a73a5890a0fa70af5c44e45c WatchSource:0}: Error finding container 3eca61f088f9ff743f2af52e51dbd367624b5fa4a73a5890a0fa70af5c44e45c: Status 404 returned error can't find the container with id 3eca61f088f9ff743f2af52e51dbd367624b5fa4a73a5890a0fa70af5c44e45c Oct 30 19:54:59.864171 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.863525 5047 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:54:59.865170 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:54:59.865146 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj"] Oct 30 19:55:00.315424 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:00.315395 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" Oct 30 19:55:00.315608 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:00.315465 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" Oct 30 19:55:00.453086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:00.452135 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" event={"ID":"70395564-6b94-4b5c-af03-06efe90cb46a","Type":"ContainerStarted","Data":"5761efb22616f23972db4b28d1667f2f1f75d323f205f02b711d6b4c458997cf"} Oct 30 19:55:00.453086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:00.452212 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" event={"ID":"70395564-6b94-4b5c-af03-06efe90cb46a","Type":"ContainerStarted","Data":"3eca61f088f9ff743f2af52e51dbd367624b5fa4a73a5890a0fa70af5c44e45c"} Oct 30 19:55:00.453086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:00.452370 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 19:55:00.477789 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:00.477751 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 19:55:00.478042 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:00.477809 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" podUID="c0cd76c6-9b67-4719-9d13-82315cd9f77a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 19:55:00.491725 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:00.491658 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" podStartSLOduration=1.159803358 podStartE2EDuration="1.491637012s" totalImagesPullingTime="331.833654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:00.487975371 +0000 UTC m=+3053.562430742" watchObservedRunningTime="2026-10-30 19:55:00.491637012 +0000 UTC m=+3053.566092374" Oct 30 19:55:01.484272 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:01.484239 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:55:01.484765 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:01.484744 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:55:01.485553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:01.485517 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 19:55:06.465839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:06.465806 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 19:55:09.602781 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:55:09.602737 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:55:24.827280 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:24.827243 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5"] Oct 30 19:55:24.835350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:24.835322 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" Oct 30 19:55:24.856488 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:24.856451 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5"] Oct 30 19:55:24.858954 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:24.858909 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1b36ad-dockercfg-vg5nk\"" Oct 30 19:55:24.863000 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:24.861759 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1b36ad\"" Oct 30 19:55:24.954738 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:24.954706 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjdd7\" (UniqueName: \"kubernetes.io/projected/69575b4d-726e-49dc-b0d8-50be11ea88db-kube-api-access-jjdd7\") pod \"payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5\" (UID: \"69575b4d-726e-49dc-b0d8-50be11ea88db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" Oct 30 19:55:24.954917 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:24.954898 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69575b4d-726e-49dc-b0d8-50be11ea88db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5\" (UID: \"69575b4d-726e-49dc-b0d8-50be11ea88db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" Oct 30 19:55:25.061817 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:25.061775 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69575b4d-726e-49dc-b0d8-50be11ea88db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5\" (UID: \"69575b4d-726e-49dc-b0d8-50be11ea88db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" Oct 30 19:55:25.062246 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:25.062221 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjdd7\" (UniqueName: \"kubernetes.io/projected/69575b4d-726e-49dc-b0d8-50be11ea88db-kube-api-access-jjdd7\") pod \"payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5\" (UID: \"69575b4d-726e-49dc-b0d8-50be11ea88db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" Oct 30 19:55:25.063720 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:25.063694 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69575b4d-726e-49dc-b0d8-50be11ea88db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5\" (UID: \"69575b4d-726e-49dc-b0d8-50be11ea88db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" Oct 30 19:55:25.079473 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:25.079443 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjdd7\" (UniqueName: \"kubernetes.io/projected/69575b4d-726e-49dc-b0d8-50be11ea88db-kube-api-access-jjdd7\") pod \"payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5\" (UID: \"69575b4d-726e-49dc-b0d8-50be11ea88db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" Oct 30 19:55:25.171374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:25.171344 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" Oct 30 19:55:25.496928 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:25.496897 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5"] Oct 30 19:55:25.497416 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:55:25.497391 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69575b4d_726e_49dc_b0d8_50be11ea88db.slice/crio-8e9300f0ae7bdeba9bc1a4c90614557c3a9432de630525a40b9ddd9a306aa990 WatchSource:0}: Error finding container 8e9300f0ae7bdeba9bc1a4c90614557c3a9432de630525a40b9ddd9a306aa990: Status 404 returned error can't find the container with id 8e9300f0ae7bdeba9bc1a4c90614557c3a9432de630525a40b9ddd9a306aa990 Oct 30 19:55:25.763860 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:25.763746 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" event={"ID":"69575b4d-726e-49dc-b0d8-50be11ea88db","Type":"ContainerStarted","Data":"2d673fd2d9008f40ffce6c82d342b21a957680c0e235224ae57c199464a0f2cf"} Oct 30 19:55:25.763860 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:25.763836 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" event={"ID":"69575b4d-726e-49dc-b0d8-50be11ea88db","Type":"ContainerStarted","Data":"8e9300f0ae7bdeba9bc1a4c90614557c3a9432de630525a40b9ddd9a306aa990"} Oct 30 19:55:25.764097 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:25.763869 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" Oct 30 19:55:25.799984 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:25.799930 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" podStartSLOduration=1.799915705 podStartE2EDuration="1.799915705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:25.797663881 +0000 UTC m=+3078.872119275" watchObservedRunningTime="2026-10-30 19:55:25.799915705 +0000 UTC m=+3078.874371109" Oct 30 19:55:36.774130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:36.774098 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" Oct 30 19:55:38.911534 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:38.911495 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z"] Oct 30 19:55:38.912093 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:38.911754 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" podUID="3824e276-5eec-4cea-a06b-c81cc9853ed4" containerName="istio-proxy" containerID="cri-o://b4face593bc04bd1e82c381924a391418f27208aaff6b94121f0db36eced50a8" gracePeriod=30 Oct 30 19:55:39.602113 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:55:39.602079 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:55:43.897070 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:43.897030 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": read tcp 10.132.0.2:47952->10.132.0.200:15021: read: connection reset by peer" start-of-body= Oct 30 19:55:43.899450 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:43.897106 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" podUID="c4e3fe64-b5fe-453a-ae2b-7dec205c7931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": read tcp 10.132.0.2:47952->10.132.0.200:15021: read: connection reset by peer" Oct 30 19:55:44.027946 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.027919 5047 generic.go:356] "Generic (PLEG): container finished" podID="c4e3fe64-b5fe-453a-ae2b-7dec205c7931" containerID="1c2ff8d309070abb77eff14bdeabca03ab01bc4f16059298cba917c96dfc8659" exitCode=0 Oct 30 19:55:44.028096 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.027987 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" event={"ID":"c4e3fe64-b5fe-453a-ae2b-7dec205c7931","Type":"ContainerDied","Data":"1c2ff8d309070abb77eff14bdeabca03ab01bc4f16059298cba917c96dfc8659"} Oct 30 19:55:44.028545 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.028523 5047 scope.go:122] "RemoveContainer" containerID="1c2ff8d309070abb77eff14bdeabca03ab01bc4f16059298cba917c96dfc8659" Oct 30 19:55:44.030521 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.030494 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:55:44.030594 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.030561 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:55:44.030632 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.030598 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:55:44.282790 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.282752 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:55:44.305805 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.305769 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-token\" (UniqueName: \"kubernetes.io/projected/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-token\") pod \"3824e276-5eec-4cea-a06b-c81cc9853ed4\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " Oct 30 19:55:44.305805 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.305812 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-podinfo\") pod \"3824e276-5eec-4cea-a06b-c81cc9853ed4\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " Oct 30 19:55:44.306141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.305845 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3824e276-5eec-4cea-a06b-c81cc9853ed4-kube-api-access-qh74z\" (UniqueName: \"kubernetes.io/projected/3824e276-5eec-4cea-a06b-c81cc9853ed4-kube-api-access-qh74z\") pod \"3824e276-5eec-4cea-a06b-c81cc9853ed4\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " Oct 30 19:55:44.306141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.305880 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-data\") pod \"3824e276-5eec-4cea-a06b-c81cc9853ed4\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " Oct 30 19:55:44.306141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.305938 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-workload-socket\") pod \"3824e276-5eec-4cea-a06b-c81cc9853ed4\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " Oct 30 19:55:44.306320 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.306252 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-workload-socket" pod "3824e276-5eec-4cea-a06b-c81cc9853ed4" (UID: "3824e276-5eec-4cea-a06b-c81cc9853ed4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:44.306320 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.306274 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-data" pod "3824e276-5eec-4cea-a06b-c81cc9853ed4" (UID: "3824e276-5eec-4cea-a06b-c81cc9853ed4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:44.306432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.306363 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-envoy\") pod \"3824e276-5eec-4cea-a06b-c81cc9853ed4\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " Oct 30 19:55:44.306432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.306422 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3824e276-5eec-4cea-a06b-c81cc9853ed4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3824e276-5eec-4cea-a06b-c81cc9853ed4-istiod-ca-cert\") pod \"3824e276-5eec-4cea-a06b-c81cc9853ed4\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " Oct 30 19:55:44.306582 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.306564 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-credential-socket\") pod \"3824e276-5eec-4cea-a06b-c81cc9853ed4\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " Oct 30 19:55:44.306646 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.306627 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-workload-certs\") pod \"3824e276-5eec-4cea-a06b-c81cc9853ed4\" (UID: \"3824e276-5eec-4cea-a06b-c81cc9853ed4\") " Oct 30 19:55:44.306810 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.306771 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-credential-socket" pod "3824e276-5eec-4cea-a06b-c81cc9853ed4" (UID: "3824e276-5eec-4cea-a06b-c81cc9853ed4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:44.307075 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.306977 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-workload-certs" pod "3824e276-5eec-4cea-a06b-c81cc9853ed4" (UID: "3824e276-5eec-4cea-a06b-c81cc9853ed4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:44.307232 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.307206 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3824e276-5eec-4cea-a06b-c81cc9853ed4-istiod-ca-cert" pod "3824e276-5eec-4cea-a06b-c81cc9853ed4" (UID: "3824e276-5eec-4cea-a06b-c81cc9853ed4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:44.308456 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.307589 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:55:44.308456 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.307609 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:55:44.308456 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.307624 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3824e276-5eec-4cea-a06b-c81cc9853ed4-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:55:44.308456 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.307639 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:55:44.308456 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.307654 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:55:44.308820 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.308691 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-token" pod "3824e276-5eec-4cea-a06b-c81cc9853ed4" (UID: "3824e276-5eec-4cea-a06b-c81cc9853ed4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:44.309045 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.308980 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-envoy" pod "3824e276-5eec-4cea-a06b-c81cc9853ed4" (UID: "3824e276-5eec-4cea-a06b-c81cc9853ed4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:44.309276 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.309258 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-podinfo" pod "3824e276-5eec-4cea-a06b-c81cc9853ed4" (UID: "3824e276-5eec-4cea-a06b-c81cc9853ed4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:55:44.309580 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.309561 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3824e276-5eec-4cea-a06b-c81cc9853ed4-kube-api-access-qh74z" pod "3824e276-5eec-4cea-a06b-c81cc9853ed4" (UID: "3824e276-5eec-4cea-a06b-c81cc9853ed4"). InnerVolumeSpecName "kube-api-access-qh74z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:44.409442 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.409403 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:55:44.409442 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.409439 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:55:44.409652 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.409454 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh74z\" (UniqueName: \"kubernetes.io/projected/3824e276-5eec-4cea-a06b-c81cc9853ed4-kube-api-access-qh74z\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:55:44.409652 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:44.409469 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3824e276-5eec-4cea-a06b-c81cc9853ed4-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:55:45.121274 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:45.121215 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" event={"ID":"c4e3fe64-b5fe-453a-ae2b-7dec205c7931","Type":"ContainerStarted","Data":"3095578aa737bb053919356564b3707ea12c112e6dd8875bc3b09ec918e46151"} Oct 30 19:55:45.123854 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:45.123825 5047 generic.go:356] "Generic (PLEG): container finished" podID="3824e276-5eec-4cea-a06b-c81cc9853ed4" containerID="b4face593bc04bd1e82c381924a391418f27208aaff6b94121f0db36eced50a8" exitCode=0 Oct 30 19:55:45.123971 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:45.123904 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" Oct 30 19:55:45.124039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:45.123899 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" event={"ID":"3824e276-5eec-4cea-a06b-c81cc9853ed4","Type":"ContainerDied","Data":"b4face593bc04bd1e82c381924a391418f27208aaff6b94121f0db36eced50a8"} Oct 30 19:55:45.124086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:45.124060 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z" event={"ID":"3824e276-5eec-4cea-a06b-c81cc9853ed4","Type":"ContainerDied","Data":"9b250c5f4d62427c83fd397dff75083d2640add35f8bfbed0b389b04974db894"} Oct 30 19:55:45.124135 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:45.124085 5047 scope.go:122] "RemoveContainer" containerID="b4face593bc04bd1e82c381924a391418f27208aaff6b94121f0db36eced50a8" Oct 30 19:55:45.160128 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:45.146189 5047 scope.go:122] "RemoveContainer" containerID="b4face593bc04bd1e82c381924a391418f27208aaff6b94121f0db36eced50a8" Oct 30 19:55:45.160563 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:55:45.160518 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4face593bc04bd1e82c381924a391418f27208aaff6b94121f0db36eced50a8\": container with ID starting with b4face593bc04bd1e82c381924a391418f27208aaff6b94121f0db36eced50a8 not found: ID does not exist" containerID="b4face593bc04bd1e82c381924a391418f27208aaff6b94121f0db36eced50a8" Oct 30 19:55:45.160683 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:45.160601 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4face593bc04bd1e82c381924a391418f27208aaff6b94121f0db36eced50a8"} err="failed to get container status \"b4face593bc04bd1e82c381924a391418f27208aaff6b94121f0db36eced50a8\": rpc error: code = NotFound desc = could not find container \"b4face593bc04bd1e82c381924a391418f27208aaff6b94121f0db36eced50a8\": container with ID starting with b4face593bc04bd1e82c381924a391418f27208aaff6b94121f0db36eced50a8 not found: ID does not exist" Oct 30 19:55:45.204631 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:45.204541 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z"] Oct 30 19:55:45.209397 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:45.209373 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-r9r2z"] Oct 30 19:55:45.682466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:45.682423 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:55:45.682681 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:45.682623 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 19:55:45.682760 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:45.682676 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" podUID="c4e3fe64-b5fe-453a-ae2b-7dec205c7931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 19:55:46.683998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:46.681990 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 19:55:46.683998 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:46.682091 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" podUID="c4e3fe64-b5fe-453a-ae2b-7dec205c7931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 19:55:47.073466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:47.073389 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3824e276-5eec-4cea-a06b-c81cc9853ed4" path="/var/lib/kubelet/pods/3824e276-5eec-4cea-a06b-c81cc9853ed4/volumes" Oct 30 19:55:47.682263 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:47.682227 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 19:55:47.682419 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:47.682292 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" podUID="c4e3fe64-b5fe-453a-ae2b-7dec205c7931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 19:55:48.682860 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:48.682824 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 19:55:48.683350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:48.682882 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" podUID="c4e3fe64-b5fe-453a-ae2b-7dec205c7931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 19:55:49.682307 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:49.682272 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 19:55:49.682495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:49.682334 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" podUID="c4e3fe64-b5fe-453a-ae2b-7dec205c7931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 19:55:50.790194 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:50.790159 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:55:50.791241 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:50.791206 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:55:50.792120 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:50.792096 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 19:55:54.701238 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:54.699673 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc"] Oct 30 19:55:54.701848 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:54.701820 5047 cpu_manager.go:414] "containerMap: removing container" podUID="3824e276-5eec-4cea-a06b-c81cc9853ed4" containerName="istio-proxy" Oct 30 19:55:54.702639 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:54.701963 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3824e276-5eec-4cea-a06b-c81cc9853ed4" containerName="istio-proxy" Oct 30 19:55:54.703105 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:54.703083 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="3824e276-5eec-4cea-a06b-c81cc9853ed4" containerName="istio-proxy" Oct 30 19:55:54.710222 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:54.710202 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" Oct 30 19:55:54.776161 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:54.741436 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b867838d-f9a0-45e6-abd3-1d89dabbe6ca-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc\" (UID: \"b867838d-f9a0-45e6-abd3-1d89dabbe6ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" Oct 30 19:55:54.798908 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:54.798873 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmm86\" (UniqueName: \"kubernetes.io/projected/b867838d-f9a0-45e6-abd3-1d89dabbe6ca-kube-api-access-bmm86\") pod \"payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc\" (UID: \"b867838d-f9a0-45e6-abd3-1d89dabbe6ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" Oct 30 19:55:54.799585 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:54.756207 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-315ea2\"" Oct 30 19:55:54.799730 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:54.794769 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc"] Oct 30 19:55:54.799900 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:54.755978 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-315ea2-dockercfg-l9xnd\"" Oct 30 19:55:54.901135 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:54.901097 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b867838d-f9a0-45e6-abd3-1d89dabbe6ca-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc\" (UID: \"b867838d-f9a0-45e6-abd3-1d89dabbe6ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" Oct 30 19:55:54.901135 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:54.901141 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmm86\" (UniqueName: \"kubernetes.io/projected/b867838d-f9a0-45e6-abd3-1d89dabbe6ca-kube-api-access-bmm86\") pod \"payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc\" (UID: \"b867838d-f9a0-45e6-abd3-1d89dabbe6ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" Oct 30 19:55:54.901934 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:54.901910 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b867838d-f9a0-45e6-abd3-1d89dabbe6ca-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc\" (UID: \"b867838d-f9a0-45e6-abd3-1d89dabbe6ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" Oct 30 19:55:54.918769 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:54.918737 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmm86\" (UniqueName: \"kubernetes.io/projected/b867838d-f9a0-45e6-abd3-1d89dabbe6ca-kube-api-access-bmm86\") pod \"payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc\" (UID: \"b867838d-f9a0-45e6-abd3-1d89dabbe6ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" Oct 30 19:55:55.058862 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:55.058770 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" Oct 30 19:55:55.359596 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:55.359567 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc"] Oct 30 19:55:55.360250 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:55:55.360226 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb867838d_f9a0_45e6_abd3_1d89dabbe6ca.slice/crio-04eb4fdce1de9053cb93a62d3d00c659330ff557646f2447d5b8ba6190e09266 WatchSource:0}: Error finding container 04eb4fdce1de9053cb93a62d3d00c659330ff557646f2447d5b8ba6190e09266: Status 404 returned error can't find the container with id 04eb4fdce1de9053cb93a62d3d00c659330ff557646f2447d5b8ba6190e09266 Oct 30 19:55:56.257782 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:56.257742 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" event={"ID":"b867838d-f9a0-45e6-abd3-1d89dabbe6ca","Type":"ContainerStarted","Data":"12929bf02542ecce769c607d3254b4040402d54e6832cb6281e1fa73d7f090c3"} Oct 30 19:55:56.258464 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:56.258428 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" event={"ID":"b867838d-f9a0-45e6-abd3-1d89dabbe6ca","Type":"ContainerStarted","Data":"04eb4fdce1de9053cb93a62d3d00c659330ff557646f2447d5b8ba6190e09266"} Oct 30 19:55:56.259608 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:56.259586 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" Oct 30 19:55:56.372029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:55:56.366352 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" podStartSLOduration=2.366329663 podStartE2EDuration="2.366329663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:56.306602017 +0000 UTC m=+3109.381057384" watchObservedRunningTime="2026-10-30 19:55:56.366329663 +0000 UTC m=+3109.440785023" Oct 30 19:56:02.175307 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:02.175262 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9"] Oct 30 19:56:02.207104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:02.207068 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w"] Oct 30 19:56:07.176110 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:07.176061 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" podUID="d3d82371-f264-41cc-b5d1-5cb5f5b35086" containerName="payload-processing" containerID="cri-o://fbb17375dbca5989e7bdd73409408930492eb8c576daa129df6a2bb86abab1f1" gracePeriod=25 Oct 30 19:56:07.208251 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:07.208201 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" podUID="6c9f0447-ecc0-4d98-95fe-3d11df134549" containerName="payload-pre-processing" containerID="cri-o://7afc270683388ea1a265556bfa460f56d0ff3e52042e6ac16171018246a225ba" gracePeriod=25 Oct 30 19:56:08.277814 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:08.277784 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" Oct 30 19:56:09.602545 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:56:09.602508 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:56:12.844329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:12.844293 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" podUID="6c9f0447-ecc0-4d98-95fe-3d11df134549" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:56:12.844734 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:12.844341 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" podUID="d3d82371-f264-41cc-b5d1-5cb5f5b35086" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:56:20.259348 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:20.259310 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk"] Oct 30 19:56:22.844619 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:22.844574 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" podUID="d3d82371-f264-41cc-b5d1-5cb5f5b35086" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:56:22.845161 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:22.844666 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" podUID="6c9f0447-ecc0-4d98-95fe-3d11df134549" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:56:23.691260 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:23.691223 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld"] Oct 30 19:56:23.691508 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:23.691485 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" podUID="9f350d91-ec30-4852-aa89-ad093cb6606a" containerName="maas-api" containerID="cri-o://128323fa0fa0abe3e3d21feddc9675a7f0a3341b3963a682162c7c1d924fd81f" gracePeriod=30 Oct 30 19:56:23.760123 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:23.760089 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" podUID="9f350d91-ec30-4852-aa89-ad093cb6606a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.161:8443/health\": dial tcp 10.132.0.161:8443: connect: connection refused" Oct 30 19:56:24.287060 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.278657 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:56:24.363227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.363196 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f350d91-ec30-4852-aa89-ad093cb6606a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f350d91-ec30-4852-aa89-ad093cb6606a-metrics-certs\") pod \"9f350d91-ec30-4852-aa89-ad093cb6606a\" (UID: \"9f350d91-ec30-4852-aa89-ad093cb6606a\") " Oct 30 19:56:24.363683 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.363660 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f350d91-ec30-4852-aa89-ad093cb6606a-kube-api-access-sc7kn\" (UniqueName: \"kubernetes.io/projected/9f350d91-ec30-4852-aa89-ad093cb6606a-kube-api-access-sc7kn\") pod \"9f350d91-ec30-4852-aa89-ad093cb6606a\" (UID: \"9f350d91-ec30-4852-aa89-ad093cb6606a\") " Oct 30 19:56:24.364370 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.364345 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f350d91-ec30-4852-aa89-ad093cb6606a-openshift-service-ca" pod "9f350d91-ec30-4852-aa89-ad093cb6606a" (UID: "9f350d91-ec30-4852-aa89-ad093cb6606a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:24.364499 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.363998 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f350d91-ec30-4852-aa89-ad093cb6606a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f350d91-ec30-4852-aa89-ad093cb6606a-openshift-service-ca\") pod \"9f350d91-ec30-4852-aa89-ad093cb6606a\" (UID: \"9f350d91-ec30-4852-aa89-ad093cb6606a\") " Oct 30 19:56:24.364499 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.364484 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f350d91-ec30-4852-aa89-ad093cb6606a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f350d91-ec30-4852-aa89-ad093cb6606a-maas-api-tls\") pod \"9f350d91-ec30-4852-aa89-ad093cb6606a\" (UID: \"9f350d91-ec30-4852-aa89-ad093cb6606a\") " Oct 30 19:56:24.380455 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.369188 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f350d91-ec30-4852-aa89-ad093cb6606a-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:24.380455 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.380336 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f350d91-ec30-4852-aa89-ad093cb6606a-maas-api-tls" pod "9f350d91-ec30-4852-aa89-ad093cb6606a" (UID: "9f350d91-ec30-4852-aa89-ad093cb6606a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:24.382461 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.382424 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f350d91-ec30-4852-aa89-ad093cb6606a-metrics-certs" pod "9f350d91-ec30-4852-aa89-ad093cb6606a" (UID: "9f350d91-ec30-4852-aa89-ad093cb6606a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:24.384247 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.384220 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f350d91-ec30-4852-aa89-ad093cb6606a-kube-api-access-sc7kn" pod "9f350d91-ec30-4852-aa89-ad093cb6606a" (UID: "9f350d91-ec30-4852-aa89-ad093cb6606a"). InnerVolumeSpecName "kube-api-access-sc7kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:24.471257 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.471216 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f350d91-ec30-4852-aa89-ad093cb6606a-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:24.471257 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.471258 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f350d91-ec30-4852-aa89-ad093cb6606a-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:24.471517 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.471275 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc7kn\" (UniqueName: \"kubernetes.io/projected/9f350d91-ec30-4852-aa89-ad093cb6606a-kube-api-access-sc7kn\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:24.713713 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.713614 5047 generic.go:356] "Generic (PLEG): container finished" podID="9f350d91-ec30-4852-aa89-ad093cb6606a" containerID="128323fa0fa0abe3e3d21feddc9675a7f0a3341b3963a682162c7c1d924fd81f" exitCode=0 Oct 30 19:56:24.713713 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.713669 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" Oct 30 19:56:24.713713 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.713699 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" event={"ID":"9f350d91-ec30-4852-aa89-ad093cb6606a","Type":"ContainerDied","Data":"128323fa0fa0abe3e3d21feddc9675a7f0a3341b3963a682162c7c1d924fd81f"} Oct 30 19:56:24.714033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.713753 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld" event={"ID":"9f350d91-ec30-4852-aa89-ad093cb6606a","Type":"ContainerDied","Data":"97f7b9daeeb5ce1420a19040b1e54be3a8f1abf974412cd020fcab438452dfaf"} Oct 30 19:56:24.714033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.713775 5047 scope.go:122] "RemoveContainer" containerID="128323fa0fa0abe3e3d21feddc9675a7f0a3341b3963a682162c7c1d924fd81f" Oct 30 19:56:24.753503 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.753475 5047 scope.go:122] "RemoveContainer" containerID="128323fa0fa0abe3e3d21feddc9675a7f0a3341b3963a682162c7c1d924fd81f" Oct 30 19:56:24.754556 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:56:24.754521 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"128323fa0fa0abe3e3d21feddc9675a7f0a3341b3963a682162c7c1d924fd81f\": container with ID starting with 128323fa0fa0abe3e3d21feddc9675a7f0a3341b3963a682162c7c1d924fd81f not found: ID does not exist" containerID="128323fa0fa0abe3e3d21feddc9675a7f0a3341b3963a682162c7c1d924fd81f" Oct 30 19:56:24.754717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.754567 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128323fa0fa0abe3e3d21feddc9675a7f0a3341b3963a682162c7c1d924fd81f"} err="failed to get container status \"128323fa0fa0abe3e3d21feddc9675a7f0a3341b3963a682162c7c1d924fd81f\": rpc error: code = NotFound desc = could not find container \"128323fa0fa0abe3e3d21feddc9675a7f0a3341b3963a682162c7c1d924fd81f\": container with ID starting with 128323fa0fa0abe3e3d21feddc9675a7f0a3341b3963a682162c7c1d924fd81f not found: ID does not exist" Oct 30 19:56:24.806626 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.805197 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld"] Oct 30 19:56:24.810328 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:24.810295 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f44f74-55b8c66c7b-qxwld"] Oct 30 19:56:25.051355 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:25.051280 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f350d91-ec30-4852-aa89-ad093cb6606a" path="/var/lib/kubelet/pods/9f350d91-ec30-4852-aa89-ad093cb6606a/volumes" Oct 30 19:56:25.844850 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:25.844811 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g"] Oct 30 19:56:25.846168 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:25.846143 5047 cpu_manager.go:414] "containerMap: removing container" podUID="9f350d91-ec30-4852-aa89-ad093cb6606a" containerName="maas-api" Oct 30 19:56:25.846241 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:25.846170 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f350d91-ec30-4852-aa89-ad093cb6606a" containerName="maas-api" Oct 30 19:56:25.846407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:25.846392 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f350d91-ec30-4852-aa89-ad093cb6606a" containerName="maas-api" Oct 30 19:56:25.850906 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:25.850883 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:25.882826 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:25.882797 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g"] Oct 30 19:56:25.995350 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:25.995313 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-workload-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:25.995504 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:25.995361 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-envoy\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:25.995504 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:25.995443 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-podinfo\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:25.995637 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:25.995568 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istiod-ca-cert\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:25.995750 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:25.995717 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-credential-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:25.995895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:25.995776 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-workload-certs\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:25.995895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:25.995846 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dsfp\" (UniqueName: \"kubernetes.io/projected/01aec079-7f44-4c06-98a5-cb46a0efeb3b-kube-api-access-4dsfp\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:25.996055 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:25.995935 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-data\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:25.996055 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:25.995975 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-token\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.097740 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.097674 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-credential-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.097740 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.097707 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-workload-certs\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.097740 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.097740 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dsfp\" (UniqueName: \"kubernetes.io/projected/01aec079-7f44-4c06-98a5-cb46a0efeb3b-kube-api-access-4dsfp\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.098046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.097776 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-data\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.098046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.097809 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-token\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.098046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.097907 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-workload-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.098046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.097941 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-envoy\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.098046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.097985 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-podinfo\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.098335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.098084 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istiod-ca-cert\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.098335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.098233 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-workload-certs\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.098335 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.098311 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-credential-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.098507 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.098392 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-workload-socket\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.098627 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.098607 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-data\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.098785 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.098767 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istiod-ca-cert\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.100340 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.100318 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-envoy\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.100587 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.100559 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-podinfo\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.114421 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.114398 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-token\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.114793 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.114762 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dsfp\" (UniqueName: \"kubernetes.io/projected/01aec079-7f44-4c06-98a5-cb46a0efeb3b-kube-api-access-4dsfp\") pod \"e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.166318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.166294 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:26.479099 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:56:26.479072 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01aec079_7f44_4c06_98a5_cb46a0efeb3b.slice/crio-6352d38eb20c1efb1601a8f6d4b34d2c648ebc3dcc8fc8429de3d8ad3a9973a6 WatchSource:0}: Error finding container 6352d38eb20c1efb1601a8f6d4b34d2c648ebc3dcc8fc8429de3d8ad3a9973a6: Status 404 returned error can't find the container with id 6352d38eb20c1efb1601a8f6d4b34d2c648ebc3dcc8fc8429de3d8ad3a9973a6 Oct 30 19:56:26.479784 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.479757 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g"] Oct 30 19:56:26.482253 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.482217 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:56:26.482346 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.482286 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:56:26.482346 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.482312 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:56:26.733584 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.733508 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" event={"ID":"01aec079-7f44-4c06-98a5-cb46a0efeb3b","Type":"ContainerStarted","Data":"c5e4b7fa79f26a6f78fee3fae951ca6b47163ba1f6f30d9092f5e6575eda8ec3"} Oct 30 19:56:26.733584 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.733563 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" event={"ID":"01aec079-7f44-4c06-98a5-cb46a0efeb3b","Type":"ContainerStarted","Data":"6352d38eb20c1efb1601a8f6d4b34d2c648ebc3dcc8fc8429de3d8ad3a9973a6"} Oct 30 19:56:26.771939 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:26.771891 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" podStartSLOduration=1.7718788239999999 podStartE2EDuration="1.771878824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:26.769742676 +0000 UTC m=+3139.844198034" watchObservedRunningTime="2026-10-30 19:56:26.771878824 +0000 UTC m=+3139.846334180" Oct 30 19:56:27.167230 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.167185 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:27.169069 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.169037 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 19:56:27.169181 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.169111 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" podUID="01aec079-7f44-4c06-98a5-cb46a0efeb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 19:56:27.596801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.584765 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" Oct 30 19:56:27.624178 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.624154 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" Oct 30 19:56:27.723631 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.723584 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9f0447-ecc0-4d98-95fe-3d11df134549-kube-api-access-249sj\" (UniqueName: \"kubernetes.io/projected/6c9f0447-ecc0-4d98-95fe-3d11df134549-kube-api-access-249sj\") pod \"6c9f0447-ecc0-4d98-95fe-3d11df134549\" (UID: \"6c9f0447-ecc0-4d98-95fe-3d11df134549\") " Oct 30 19:56:27.723801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.723734 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c9f0447-ecc0-4d98-95fe-3d11df134549-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9f0447-ecc0-4d98-95fe-3d11df134549-plugins-config-volume\") pod \"6c9f0447-ecc0-4d98-95fe-3d11df134549\" (UID: \"6c9f0447-ecc0-4d98-95fe-3d11df134549\") " Oct 30 19:56:27.723861 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.723842 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d82371-f264-41cc-b5d1-5cb5f5b35086-kube-api-access-vfw6n\" (UniqueName: \"kubernetes.io/projected/d3d82371-f264-41cc-b5d1-5cb5f5b35086-kube-api-access-vfw6n\") pod \"d3d82371-f264-41cc-b5d1-5cb5f5b35086\" (UID: \"d3d82371-f264-41cc-b5d1-5cb5f5b35086\") " Oct 30 19:56:27.723916 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.723900 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d82371-f264-41cc-b5d1-5cb5f5b35086-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d82371-f264-41cc-b5d1-5cb5f5b35086-plugins-config-volume\") pod \"d3d82371-f264-41cc-b5d1-5cb5f5b35086\" (UID: \"d3d82371-f264-41cc-b5d1-5cb5f5b35086\") " Oct 30 19:56:27.724206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.724175 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9f0447-ecc0-4d98-95fe-3d11df134549-plugins-config-volume" pod "6c9f0447-ecc0-4d98-95fe-3d11df134549" (UID: "6c9f0447-ecc0-4d98-95fe-3d11df134549"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:27.724321 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.724291 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d82371-f264-41cc-b5d1-5cb5f5b35086-plugins-config-volume" pod "d3d82371-f264-41cc-b5d1-5cb5f5b35086" (UID: "d3d82371-f264-41cc-b5d1-5cb5f5b35086"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:27.724600 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.724579 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9f0447-ecc0-4d98-95fe-3d11df134549-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:27.724690 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.724601 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d82371-f264-41cc-b5d1-5cb5f5b35086-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:27.725862 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.725838 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9f0447-ecc0-4d98-95fe-3d11df134549-kube-api-access-249sj" pod "6c9f0447-ecc0-4d98-95fe-3d11df134549" (UID: "6c9f0447-ecc0-4d98-95fe-3d11df134549"). InnerVolumeSpecName "kube-api-access-249sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:27.726193 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.726166 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d82371-f264-41cc-b5d1-5cb5f5b35086-kube-api-access-vfw6n" pod "d3d82371-f264-41cc-b5d1-5cb5f5b35086" (UID: "d3d82371-f264-41cc-b5d1-5cb5f5b35086"). InnerVolumeSpecName "kube-api-access-vfw6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:27.773950 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.773852 5047 generic.go:356] "Generic (PLEG): container finished" podID="6c9f0447-ecc0-4d98-95fe-3d11df134549" containerID="7afc270683388ea1a265556bfa460f56d0ff3e52042e6ac16171018246a225ba" exitCode=0 Oct 30 19:56:27.773950 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.773900 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" Oct 30 19:56:27.774229 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.773986 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" event={"ID":"6c9f0447-ecc0-4d98-95fe-3d11df134549","Type":"ContainerDied","Data":"7afc270683388ea1a265556bfa460f56d0ff3e52042e6ac16171018246a225ba"} Oct 30 19:56:27.774229 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.774092 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w" event={"ID":"6c9f0447-ecc0-4d98-95fe-3d11df134549","Type":"ContainerDied","Data":"ef92f274ca99a571e3ce998145ad99dd7ca518e250a880c85fe5529f94f59529"} Oct 30 19:56:27.774229 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.774124 5047 scope.go:122] "RemoveContainer" containerID="7afc270683388ea1a265556bfa460f56d0ff3e52042e6ac16171018246a225ba" Oct 30 19:56:27.776272 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.776250 5047 generic.go:356] "Generic (PLEG): container finished" podID="d3d82371-f264-41cc-b5d1-5cb5f5b35086" containerID="fbb17375dbca5989e7bdd73409408930492eb8c576daa129df6a2bb86abab1f1" exitCode=0 Oct 30 19:56:27.776404 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.776303 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" Oct 30 19:56:27.776404 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.776295 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" event={"ID":"d3d82371-f264-41cc-b5d1-5cb5f5b35086","Type":"ContainerDied","Data":"fbb17375dbca5989e7bdd73409408930492eb8c576daa129df6a2bb86abab1f1"} Oct 30 19:56:27.776404 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.776399 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9" event={"ID":"d3d82371-f264-41cc-b5d1-5cb5f5b35086","Type":"ContainerDied","Data":"0933d5df9e19ac393cf686114aa2c1403932ae46b8aebfab1985ee668a55cab9"} Oct 30 19:56:27.787152 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.787118 5047 scope.go:122] "RemoveContainer" containerID="7afc270683388ea1a265556bfa460f56d0ff3e52042e6ac16171018246a225ba" Oct 30 19:56:27.787551 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:56:27.787420 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7afc270683388ea1a265556bfa460f56d0ff3e52042e6ac16171018246a225ba\": container with ID starting with 7afc270683388ea1a265556bfa460f56d0ff3e52042e6ac16171018246a225ba not found: ID does not exist" containerID="7afc270683388ea1a265556bfa460f56d0ff3e52042e6ac16171018246a225ba" Oct 30 19:56:27.787551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.787446 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7afc270683388ea1a265556bfa460f56d0ff3e52042e6ac16171018246a225ba"} err="failed to get container status \"7afc270683388ea1a265556bfa460f56d0ff3e52042e6ac16171018246a225ba\": rpc error: code = NotFound desc = could not find container \"7afc270683388ea1a265556bfa460f56d0ff3e52042e6ac16171018246a225ba\": container with ID starting with 7afc270683388ea1a265556bfa460f56d0ff3e52042e6ac16171018246a225ba not found: ID does not exist" Oct 30 19:56:27.787551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.787462 5047 scope.go:122] "RemoveContainer" containerID="fbb17375dbca5989e7bdd73409408930492eb8c576daa129df6a2bb86abab1f1" Oct 30 19:56:27.797384 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.797369 5047 scope.go:122] "RemoveContainer" containerID="fbb17375dbca5989e7bdd73409408930492eb8c576daa129df6a2bb86abab1f1" Oct 30 19:56:27.797637 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:56:27.797614 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb17375dbca5989e7bdd73409408930492eb8c576daa129df6a2bb86abab1f1\": container with ID starting with fbb17375dbca5989e7bdd73409408930492eb8c576daa129df6a2bb86abab1f1 not found: ID does not exist" containerID="fbb17375dbca5989e7bdd73409408930492eb8c576daa129df6a2bb86abab1f1" Oct 30 19:56:27.797768 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.797648 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb17375dbca5989e7bdd73409408930492eb8c576daa129df6a2bb86abab1f1"} err="failed to get container status \"fbb17375dbca5989e7bdd73409408930492eb8c576daa129df6a2bb86abab1f1\": rpc error: code = NotFound desc = could not find container \"fbb17375dbca5989e7bdd73409408930492eb8c576daa129df6a2bb86abab1f1\": container with ID starting with fbb17375dbca5989e7bdd73409408930492eb8c576daa129df6a2bb86abab1f1 not found: ID does not exist" Oct 30 19:56:27.825834 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.825813 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfw6n\" (UniqueName: \"kubernetes.io/projected/d3d82371-f264-41cc-b5d1-5cb5f5b35086-kube-api-access-vfw6n\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:27.825946 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.825836 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-249sj\" (UniqueName: \"kubernetes.io/projected/6c9f0447-ecc0-4d98-95fe-3d11df134549-kube-api-access-249sj\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:27.831484 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.831463 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w"] Oct 30 19:56:27.855703 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.855665 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f44f74-58b7c8b64c-m6m9w"] Oct 30 19:56:27.874503 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.874471 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9"] Oct 30 19:56:27.878740 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:27.878714 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f44f74-7f644d6db-g2rm9"] Oct 30 19:56:28.167028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:28.166969 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 19:56:28.167219 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:28.167061 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" podUID="01aec079-7f44-4c06-98a5-cb46a0efeb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 19:56:29.070565 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:29.070529 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c9f0447-ecc0-4d98-95fe-3d11df134549" path="/var/lib/kubelet/pods/6c9f0447-ecc0-4d98-95fe-3d11df134549/volumes" Oct 30 19:56:29.070974 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:29.070956 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d82371-f264-41cc-b5d1-5cb5f5b35086" path="/var/lib/kubelet/pods/d3d82371-f264-41cc-b5d1-5cb5f5b35086/volumes" Oct 30 19:56:29.167253 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:29.167221 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 19:56:29.167434 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:29.167279 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" podUID="01aec079-7f44-4c06-98a5-cb46a0efeb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 19:56:30.167355 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:30.167322 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 19:56:30.167759 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:30.167375 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" podUID="01aec079-7f44-4c06-98a5-cb46a0efeb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 19:56:31.167099 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:31.167049 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 19:56:31.167281 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:31.167128 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" podUID="01aec079-7f44-4c06-98a5-cb46a0efeb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 19:56:31.382406 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:31.382372 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d"] Oct 30 19:56:31.422586 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:31.422490 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc"] Oct 30 19:56:31.600050 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:31.598679 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf"] Oct 30 19:56:31.600050 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:31.598993 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" podUID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" containerName="istio-proxy" containerID="cri-o://d312f4fab864e06fa964e693e03de323d06038b4fb3befd618b72b37625fdd08" gracePeriod=30 Oct 30 19:56:32.180027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:32.171372 5047 patch_prober.go:29] interesting pod/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 19:56:32.180027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:32.171435 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" podUID="01aec079-7f44-4c06-98a5-cb46a0efeb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 19:56:33.010467 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:33.010430 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r"] Oct 30 19:56:33.038726 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:33.038693 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7"] Oct 30 19:56:33.173245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:33.173214 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:33.173692 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:33.173669 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:33.174399 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:33.174379 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 19:56:34.241176 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:34.241137 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:56:34.241623 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:34.241199 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" podUID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:56:35.292673 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.292607 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" podUID="37636069-645a-4941-97fd-a2fa4b3df247" containerName="main" containerID="cri-o://21da91fc81391ec2f4951137c4c70e3e3d87acb04b5cd59578a05e94f97cb6a2" gracePeriod=45 Oct 30 19:56:35.580241 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.580214 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:56:35.701839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.701788 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37636069-645a-4941-97fd-a2fa4b3df247-kube-api-access-dczt5\" (UniqueName: \"kubernetes.io/projected/37636069-645a-4941-97fd-a2fa4b3df247-kube-api-access-dczt5\") pod \"37636069-645a-4941-97fd-a2fa4b3df247\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " Oct 30 19:56:35.701839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.701842 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-home\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-home\") pod \"37636069-645a-4941-97fd-a2fa4b3df247\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " Oct 30 19:56:35.702183 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.701976 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37636069-645a-4941-97fd-a2fa4b3df247-tls-certs\" (UniqueName: \"kubernetes.io/secret/37636069-645a-4941-97fd-a2fa4b3df247-tls-certs\") pod \"37636069-645a-4941-97fd-a2fa4b3df247\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " Oct 30 19:56:35.702183 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.702108 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-model-cache\") pod \"37636069-645a-4941-97fd-a2fa4b3df247\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " Oct 30 19:56:35.702183 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.702148 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-dshm\") pod \"37636069-645a-4941-97fd-a2fa4b3df247\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " Oct 30 19:56:35.702183 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.702169 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-home" pod "37636069-645a-4941-97fd-a2fa4b3df247" (UID: "37636069-645a-4941-97fd-a2fa4b3df247"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:35.702342 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.702225 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-tmp-dir\") pod \"37636069-645a-4941-97fd-a2fa4b3df247\" (UID: \"37636069-645a-4941-97fd-a2fa4b3df247\") " Oct 30 19:56:35.702581 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.702514 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-model-cache" pod "37636069-645a-4941-97fd-a2fa4b3df247" (UID: "37636069-645a-4941-97fd-a2fa4b3df247"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:35.702750 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.702722 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-tmp-dir" pod "37636069-645a-4941-97fd-a2fa4b3df247" (UID: "37636069-645a-4941-97fd-a2fa4b3df247"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:35.703221 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.703195 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:35.703289 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.703226 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:35.703289 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.703240 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:35.704307 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.704281 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-dshm" pod "37636069-645a-4941-97fd-a2fa4b3df247" (UID: "37636069-645a-4941-97fd-a2fa4b3df247"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:35.704430 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.704411 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37636069-645a-4941-97fd-a2fa4b3df247-tls-certs" pod "37636069-645a-4941-97fd-a2fa4b3df247" (UID: "37636069-645a-4941-97fd-a2fa4b3df247"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:35.704472 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.704450 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37636069-645a-4941-97fd-a2fa4b3df247-kube-api-access-dczt5" pod "37636069-645a-4941-97fd-a2fa4b3df247" (UID: "37636069-645a-4941-97fd-a2fa4b3df247"). InnerVolumeSpecName "kube-api-access-dczt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:35.805721 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.805634 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dczt5\" (UniqueName: \"kubernetes.io/projected/37636069-645a-4941-97fd-a2fa4b3df247-kube-api-access-dczt5\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:35.805721 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.805677 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37636069-645a-4941-97fd-a2fa4b3df247-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:35.805721 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:35.805692 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37636069-645a-4941-97fd-a2fa4b3df247-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:36.059101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.045692 5047 generic.go:356] "Generic (PLEG): container finished" podID="37636069-645a-4941-97fd-a2fa4b3df247" containerID="21da91fc81391ec2f4951137c4c70e3e3d87acb04b5cd59578a05e94f97cb6a2" exitCode=0 Oct 30 19:56:36.059101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.045981 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" event={"ID":"37636069-645a-4941-97fd-a2fa4b3df247","Type":"ContainerDied","Data":"21da91fc81391ec2f4951137c4c70e3e3d87acb04b5cd59578a05e94f97cb6a2"} Oct 30 19:56:36.059101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.046034 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" event={"ID":"37636069-645a-4941-97fd-a2fa4b3df247","Type":"ContainerDied","Data":"edb360dcad788c628c8d070f846e6b5e932cf470245ce6682674f346ba11e256"} Oct 30 19:56:36.059101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.046057 5047 scope.go:122] "RemoveContainer" containerID="21da91fc81391ec2f4951137c4c70e3e3d87acb04b5cd59578a05e94f97cb6a2" Oct 30 19:56:36.059101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.046224 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk" Oct 30 19:56:36.203833 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.203809 5047 scope.go:122] "RemoveContainer" containerID="21da91fc81391ec2f4951137c4c70e3e3d87acb04b5cd59578a05e94f97cb6a2" Oct 30 19:56:36.204189 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:56:36.204167 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21da91fc81391ec2f4951137c4c70e3e3d87acb04b5cd59578a05e94f97cb6a2\": container with ID starting with 21da91fc81391ec2f4951137c4c70e3e3d87acb04b5cd59578a05e94f97cb6a2 not found: ID does not exist" containerID="21da91fc81391ec2f4951137c4c70e3e3d87acb04b5cd59578a05e94f97cb6a2" Oct 30 19:56:36.204241 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.204200 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21da91fc81391ec2f4951137c4c70e3e3d87acb04b5cd59578a05e94f97cb6a2"} err="failed to get container status \"21da91fc81391ec2f4951137c4c70e3e3d87acb04b5cd59578a05e94f97cb6a2\": rpc error: code = NotFound desc = could not find container \"21da91fc81391ec2f4951137c4c70e3e3d87acb04b5cd59578a05e94f97cb6a2\": container with ID starting with 21da91fc81391ec2f4951137c4c70e3e3d87acb04b5cd59578a05e94f97cb6a2 not found: ID does not exist" Oct 30 19:56:36.235340 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.235301 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk"] Oct 30 19:56:36.241514 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.241477 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-aae4d8/auth-test-model-aae4d8-kserve-bcc875f57-8trgk"] Oct 30 19:56:36.408029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.405071 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" podUID="14cdb07d-1fb7-4959-baef-21ca0c5b51f1" containerName="payload-processing" containerID="cri-o://b5c30e03408f00e6322ef7454cc134976d667ebb4c330bc53073076ad4bdb91d" gracePeriod=25 Oct 30 19:56:36.412026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.410558 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n"] Oct 30 19:56:36.412026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.411765 5047 cpu_manager.go:414] "containerMap: removing container" podUID="37636069-645a-4941-97fd-a2fa4b3df247" containerName="main" Oct 30 19:56:36.412026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.411787 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37636069-645a-4941-97fd-a2fa4b3df247" containerName="main" Oct 30 19:56:36.412026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.411835 5047 cpu_manager.go:414] "containerMap: removing container" podUID="6c9f0447-ecc0-4d98-95fe-3d11df134549" containerName="payload-pre-processing" Oct 30 19:56:36.412026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.411845 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9f0447-ecc0-4d98-95fe-3d11df134549" containerName="payload-pre-processing" Oct 30 19:56:36.412026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.411888 5047 cpu_manager.go:414] "containerMap: removing container" podUID="d3d82371-f264-41cc-b5d1-5cb5f5b35086" containerName="payload-processing" Oct 30 19:56:36.412026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.411897 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d82371-f264-41cc-b5d1-5cb5f5b35086" containerName="payload-processing" Oct 30 19:56:36.420024 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.416202 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d82371-f264-41cc-b5d1-5cb5f5b35086" containerName="payload-processing" Oct 30 19:56:36.420024 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.416244 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="37636069-645a-4941-97fd-a2fa4b3df247" containerName="main" Oct 30 19:56:36.420024 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.416262 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c9f0447-ecc0-4d98-95fe-3d11df134549" containerName="payload-pre-processing" Oct 30 19:56:36.436937 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.435088 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" podUID="3475ab8d-e007-444d-baae-c24d13fd7a9d" containerName="payload-pre-processing" containerID="cri-o://7b3c74a68c63632d0dd97f1d32d4b2a8e10a66262f4244bd5971dd33db678edb" gracePeriod=25 Oct 30 19:56:36.480029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.466230 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n"] Oct 30 19:56:36.480029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.466370 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" Oct 30 19:56:36.531552 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.529228 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bf314f-dockercfg-tb77v\"" Oct 30 19:56:36.531552 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.529403 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bf314f\"" Oct 30 19:56:36.531552 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.531237 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccee9fe5-8dfb-4ee8-8e9c-5038485e3783-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n\" (UID: \"ccee9fe5-8dfb-4ee8-8e9c-5038485e3783\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" Oct 30 19:56:36.531552 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.531379 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kd5d\" (UniqueName: \"kubernetes.io/projected/ccee9fe5-8dfb-4ee8-8e9c-5038485e3783-kube-api-access-2kd5d\") pod \"payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n\" (UID: \"ccee9fe5-8dfb-4ee8-8e9c-5038485e3783\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" Oct 30 19:56:36.638227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.635256 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccee9fe5-8dfb-4ee8-8e9c-5038485e3783-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n\" (UID: \"ccee9fe5-8dfb-4ee8-8e9c-5038485e3783\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" Oct 30 19:56:36.638227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.635454 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kd5d\" (UniqueName: \"kubernetes.io/projected/ccee9fe5-8dfb-4ee8-8e9c-5038485e3783-kube-api-access-2kd5d\") pod \"payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n\" (UID: \"ccee9fe5-8dfb-4ee8-8e9c-5038485e3783\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" Oct 30 19:56:36.639175 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.639149 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccee9fe5-8dfb-4ee8-8e9c-5038485e3783-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n\" (UID: \"ccee9fe5-8dfb-4ee8-8e9c-5038485e3783\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" Oct 30 19:56:36.666378 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.666247 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kd5d\" (UniqueName: \"kubernetes.io/projected/ccee9fe5-8dfb-4ee8-8e9c-5038485e3783-kube-api-access-2kd5d\") pod \"payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n\" (UID: \"ccee9fe5-8dfb-4ee8-8e9c-5038485e3783\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" Oct 30 19:56:36.896860 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.896830 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" Oct 30 19:56:36.933330 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.933261 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:56:36.940338 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.940302 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-workload-certs\") pod \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " Oct 30 19:56:36.940471 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.940420 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-workload-socket\") pod \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " Oct 30 19:56:36.940471 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.940456 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-envoy\") pod \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " Oct 30 19:56:36.940567 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.940516 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istiod-ca-cert\") pod \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " Oct 30 19:56:36.940634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.940590 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-credential-socket\") pod \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " Oct 30 19:56:36.940634 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.940625 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-podinfo\") pod \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " Oct 30 19:56:36.940752 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.940667 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-workload-certs" pod "dec5c9c7-1994-4064-b2fe-ef9f288c53ba" (UID: "dec5c9c7-1994-4064-b2fe-ef9f288c53ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:36.940752 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.940685 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-workload-socket" pod "dec5c9c7-1994-4064-b2fe-ef9f288c53ba" (UID: "dec5c9c7-1994-4064-b2fe-ef9f288c53ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:36.940752 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.940726 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-token\" (UniqueName: \"kubernetes.io/projected/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-token\") pod \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " Oct 30 19:56:36.940921 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.940871 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-data\") pod \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " Oct 30 19:56:36.940921 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.940906 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-kube-api-access-5hlgn\" (UniqueName: \"kubernetes.io/projected/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-kube-api-access-5hlgn\") pod \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\" (UID: \"dec5c9c7-1994-4064-b2fe-ef9f288c53ba\") " Oct 30 19:56:36.941056 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.940919 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-credential-socket" pod "dec5c9c7-1994-4064-b2fe-ef9f288c53ba" (UID: "dec5c9c7-1994-4064-b2fe-ef9f288c53ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:36.941056 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.940949 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istiod-ca-cert" pod "dec5c9c7-1994-4064-b2fe-ef9f288c53ba" (UID: "dec5c9c7-1994-4064-b2fe-ef9f288c53ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:36.941272 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.941247 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-data" pod "dec5c9c7-1994-4064-b2fe-ef9f288c53ba" (UID: "dec5c9c7-1994-4064-b2fe-ef9f288c53ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:36.941656 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.941633 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:36.941753 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.941663 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:36.941753 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.941678 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:36.941753 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.941691 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:36.941753 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.941704 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:36.944723 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.944701 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-token" pod "dec5c9c7-1994-4064-b2fe-ef9f288c53ba" (UID: "dec5c9c7-1994-4064-b2fe-ef9f288c53ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:36.944837 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.944790 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-podinfo" pod "dec5c9c7-1994-4064-b2fe-ef9f288c53ba" (UID: "dec5c9c7-1994-4064-b2fe-ef9f288c53ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:56:36.944886 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.944857 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-kube-api-access-5hlgn" pod "dec5c9c7-1994-4064-b2fe-ef9f288c53ba" (UID: "dec5c9c7-1994-4064-b2fe-ef9f288c53ba"). InnerVolumeSpecName "kube-api-access-5hlgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:36.945173 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:36.945135 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-envoy" pod "dec5c9c7-1994-4064-b2fe-ef9f288c53ba" (UID: "dec5c9c7-1994-4064-b2fe-ef9f288c53ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:37.043382 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:37.043340 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:37.043382 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:37.043382 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hlgn\" (UniqueName: \"kubernetes.io/projected/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-kube-api-access-5hlgn\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:37.043677 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:37.043397 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:37.043677 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:37.043410 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec5c9c7-1994-4064-b2fe-ef9f288c53ba-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:37.065461 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:37.065425 5047 generic.go:356] "Generic (PLEG): container finished" podID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" containerID="d312f4fab864e06fa964e693e03de323d06038b4fb3befd618b72b37625fdd08" exitCode=0 Oct 30 19:56:37.065678 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:37.065516 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:56:37.065678 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:37.065438 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37636069-645a-4941-97fd-a2fa4b3df247" path="/var/lib/kubelet/pods/37636069-645a-4941-97fd-a2fa4b3df247/volumes" Oct 30 19:56:37.066557 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:37.066533 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" event={"ID":"dec5c9c7-1994-4064-b2fe-ef9f288c53ba","Type":"ContainerDied","Data":"d312f4fab864e06fa964e693e03de323d06038b4fb3befd618b72b37625fdd08"} Oct 30 19:56:37.066683 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:37.066583 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" event={"ID":"dec5c9c7-1994-4064-b2fe-ef9f288c53ba","Type":"ContainerDied","Data":"19fff0724565a3ad340924257bff05ebe5084c18c81a59ba0f31a8cf374c02bc"} Oct 30 19:56:37.066683 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:37.066615 5047 scope.go:122] "RemoveContainer" containerID="d312f4fab864e06fa964e693e03de323d06038b4fb3befd618b72b37625fdd08" Oct 30 19:56:37.083733 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:37.083714 5047 scope.go:122] "RemoveContainer" containerID="fd3e459e68032a1e32b941284916d7d31a89b7d1df2d912944e62cf9201d5d6e" Oct 30 19:56:37.094169 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:37.094149 5047 scope.go:122] "RemoveContainer" containerID="d312f4fab864e06fa964e693e03de323d06038b4fb3befd618b72b37625fdd08" Oct 30 19:56:37.094484 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:56:37.094453 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d312f4fab864e06fa964e693e03de323d06038b4fb3befd618b72b37625fdd08\": container with ID starting with d312f4fab864e06fa964e693e03de323d06038b4fb3befd618b72b37625fdd08 not found: ID does not exist" containerID="d312f4fab864e06fa964e693e03de323d06038b4fb3befd618b72b37625fdd08" Oct 30 19:56:37.094541 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:37.094490 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d312f4fab864e06fa964e693e03de323d06038b4fb3befd618b72b37625fdd08"} err="failed to get container status \"d312f4fab864e06fa964e693e03de323d06038b4fb3befd618b72b37625fdd08\": rpc error: code = NotFound desc = could not find container \"d312f4fab864e06fa964e693e03de323d06038b4fb3befd618b72b37625fdd08\": container with ID starting with d312f4fab864e06fa964e693e03de323d06038b4fb3befd618b72b37625fdd08 not found: ID does not exist" Oct 30 19:56:37.094541 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:37.094513 5047 scope.go:122] "RemoveContainer" containerID="fd3e459e68032a1e32b941284916d7d31a89b7d1df2d912944e62cf9201d5d6e" Oct 30 19:56:37.094755 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:56:37.094736 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd3e459e68032a1e32b941284916d7d31a89b7d1df2d912944e62cf9201d5d6e\": container with ID starting with fd3e459e68032a1e32b941284916d7d31a89b7d1df2d912944e62cf9201d5d6e not found: ID does not exist" containerID="fd3e459e68032a1e32b941284916d7d31a89b7d1df2d912944e62cf9201d5d6e" Oct 30 19:56:37.094849 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:37.094766 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3e459e68032a1e32b941284916d7d31a89b7d1df2d912944e62cf9201d5d6e"} err="failed to get container status \"fd3e459e68032a1e32b941284916d7d31a89b7d1df2d912944e62cf9201d5d6e\": rpc error: code = NotFound desc = could not find container \"fd3e459e68032a1e32b941284916d7d31a89b7d1df2d912944e62cf9201d5d6e\": container with ID starting with fd3e459e68032a1e32b941284916d7d31a89b7d1df2d912944e62cf9201d5d6e not found: ID does not exist" Oct 30 19:56:37.209255 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:37.209229 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n"] Oct 30 19:56:37.209357 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:56:37.209313 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccee9fe5_8dfb_4ee8_8e9c_5038485e3783.slice/crio-67511a7855eeadb40af9a8a44077ffd63d1c0065f6c3aeb5bb32b06441c39888 WatchSource:0}: Error finding container 67511a7855eeadb40af9a8a44077ffd63d1c0065f6c3aeb5bb32b06441c39888: Status 404 returned error can't find the container with id 67511a7855eeadb40af9a8a44077ffd63d1c0065f6c3aeb5bb32b06441c39888 Oct 30 19:56:38.011298 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.011196 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" podUID="14168ba7-e407-457a-8a39-b4d0c266135a" containerName="payload-processing" containerID="cri-o://4c608ef38e9e55b2ca7e5fcf3d4130b886effe03b01af298b68936f86a6d15dc" gracePeriod=25 Oct 30 19:56:38.039186 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.039146 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" podUID="d82abce5-4bc3-4a06-aa68-4e688cd1702e" containerName="payload-pre-processing" containerID="cri-o://1e2342f5bd2b75c6c6c252ebeda84410590514fd13b61e775db8e3ee725e23dc" gracePeriod=25 Oct 30 19:56:38.076841 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.076799 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" event={"ID":"ccee9fe5-8dfb-4ee8-8e9c-5038485e3783","Type":"ContainerStarted","Data":"17e388310ae3fd2f3f925280c88ed2daf1ea46965ef67c69dbe7351680102ccd"} Oct 30 19:56:38.077089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.076873 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" event={"ID":"ccee9fe5-8dfb-4ee8-8e9c-5038485e3783","Type":"ContainerStarted","Data":"67511a7855eeadb40af9a8a44077ffd63d1c0065f6c3aeb5bb32b06441c39888"} Oct 30 19:56:38.077089 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.076902 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" Oct 30 19:56:38.137541 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.137493 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" podStartSLOduration=2.137478089 podStartE2EDuration="2.137478089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:38.134471313 +0000 UTC m=+3151.208926671" watchObservedRunningTime="2026-10-30 19:56:38.137478089 +0000 UTC m=+3151.211933447" Oct 30 19:56:38.397104 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.397073 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" Oct 30 19:56:38.469974 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.469944 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" Oct 30 19:56:38.476506 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.476471 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14168ba7-e407-457a-8a39-b4d0c266135a-kube-api-access-2dgtp\" (UniqueName: \"kubernetes.io/projected/14168ba7-e407-457a-8a39-b4d0c266135a-kube-api-access-2dgtp\") pod \"14168ba7-e407-457a-8a39-b4d0c266135a\" (UID: \"14168ba7-e407-457a-8a39-b4d0c266135a\") " Oct 30 19:56:38.476673 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.476558 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14168ba7-e407-457a-8a39-b4d0c266135a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14168ba7-e407-457a-8a39-b4d0c266135a-plugins-config-volume\") pod \"14168ba7-e407-457a-8a39-b4d0c266135a\" (UID: \"14168ba7-e407-457a-8a39-b4d0c266135a\") " Oct 30 19:56:38.476926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.476898 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14168ba7-e407-457a-8a39-b4d0c266135a-plugins-config-volume" pod "14168ba7-e407-457a-8a39-b4d0c266135a" (UID: "14168ba7-e407-457a-8a39-b4d0c266135a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:38.477411 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.477381 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14168ba7-e407-457a-8a39-b4d0c266135a-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:38.478751 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.478723 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14168ba7-e407-457a-8a39-b4d0c266135a-kube-api-access-2dgtp" pod "14168ba7-e407-457a-8a39-b4d0c266135a" (UID: "14168ba7-e407-457a-8a39-b4d0c266135a"). InnerVolumeSpecName "kube-api-access-2dgtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:38.578236 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.578205 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d82abce5-4bc3-4a06-aa68-4e688cd1702e-kube-api-access-t7gpm\" (UniqueName: \"kubernetes.io/projected/d82abce5-4bc3-4a06-aa68-4e688cd1702e-kube-api-access-t7gpm\") pod \"d82abce5-4bc3-4a06-aa68-4e688cd1702e\" (UID: \"d82abce5-4bc3-4a06-aa68-4e688cd1702e\") " Oct 30 19:56:38.578431 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.578333 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d82abce5-4bc3-4a06-aa68-4e688cd1702e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d82abce5-4bc3-4a06-aa68-4e688cd1702e-plugins-config-volume\") pod \"d82abce5-4bc3-4a06-aa68-4e688cd1702e\" (UID: \"d82abce5-4bc3-4a06-aa68-4e688cd1702e\") " Oct 30 19:56:38.578791 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.578737 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d82abce5-4bc3-4a06-aa68-4e688cd1702e-plugins-config-volume" pod "d82abce5-4bc3-4a06-aa68-4e688cd1702e" (UID: "d82abce5-4bc3-4a06-aa68-4e688cd1702e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:38.578965 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.578946 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d82abce5-4bc3-4a06-aa68-4e688cd1702e-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:38.579025 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.578969 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dgtp\" (UniqueName: \"kubernetes.io/projected/14168ba7-e407-457a-8a39-b4d0c266135a-kube-api-access-2dgtp\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:38.580338 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.580312 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d82abce5-4bc3-4a06-aa68-4e688cd1702e-kube-api-access-t7gpm" pod "d82abce5-4bc3-4a06-aa68-4e688cd1702e" (UID: "d82abce5-4bc3-4a06-aa68-4e688cd1702e"). InnerVolumeSpecName "kube-api-access-t7gpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:38.680132 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:38.680104 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7gpm\" (UniqueName: \"kubernetes.io/projected/d82abce5-4bc3-4a06-aa68-4e688cd1702e-kube-api-access-t7gpm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:39.087900 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.087862 5047 generic.go:356] "Generic (PLEG): container finished" podID="d82abce5-4bc3-4a06-aa68-4e688cd1702e" containerID="1e2342f5bd2b75c6c6c252ebeda84410590514fd13b61e775db8e3ee725e23dc" exitCode=0 Oct 30 19:56:39.088360 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.087915 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" Oct 30 19:56:39.088360 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.087938 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" event={"ID":"d82abce5-4bc3-4a06-aa68-4e688cd1702e","Type":"ContainerDied","Data":"1e2342f5bd2b75c6c6c252ebeda84410590514fd13b61e775db8e3ee725e23dc"} Oct 30 19:56:39.088360 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.087992 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7" event={"ID":"d82abce5-4bc3-4a06-aa68-4e688cd1702e","Type":"ContainerDied","Data":"5d44a84d35d69fdd743bfc76b6c197f9fbdba948f68990e7f80c7c3a8264847a"} Oct 30 19:56:39.088360 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.088037 5047 scope.go:122] "RemoveContainer" containerID="1e2342f5bd2b75c6c6c252ebeda84410590514fd13b61e775db8e3ee725e23dc" Oct 30 19:56:39.089680 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.089656 5047 generic.go:356] "Generic (PLEG): container finished" podID="14168ba7-e407-457a-8a39-b4d0c266135a" containerID="4c608ef38e9e55b2ca7e5fcf3d4130b886effe03b01af298b68936f86a6d15dc" exitCode=0 Oct 30 19:56:39.089786 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.089696 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" Oct 30 19:56:39.089832 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.089788 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" event={"ID":"14168ba7-e407-457a-8a39-b4d0c266135a","Type":"ContainerDied","Data":"4c608ef38e9e55b2ca7e5fcf3d4130b886effe03b01af298b68936f86a6d15dc"} Oct 30 19:56:39.089876 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.089843 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r" event={"ID":"14168ba7-e407-457a-8a39-b4d0c266135a","Type":"ContainerDied","Data":"e8123773d997b8fba0ab210d9c15cfb35a6471164d099843db43335f7d7b5ce9"} Oct 30 19:56:39.099646 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.099628 5047 scope.go:122] "RemoveContainer" containerID="1e2342f5bd2b75c6c6c252ebeda84410590514fd13b61e775db8e3ee725e23dc" Oct 30 19:56:39.099906 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:56:39.099878 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2342f5bd2b75c6c6c252ebeda84410590514fd13b61e775db8e3ee725e23dc\": container with ID starting with 1e2342f5bd2b75c6c6c252ebeda84410590514fd13b61e775db8e3ee725e23dc not found: ID does not exist" containerID="1e2342f5bd2b75c6c6c252ebeda84410590514fd13b61e775db8e3ee725e23dc" Oct 30 19:56:39.099982 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.099912 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2342f5bd2b75c6c6c252ebeda84410590514fd13b61e775db8e3ee725e23dc"} err="failed to get container status \"1e2342f5bd2b75c6c6c252ebeda84410590514fd13b61e775db8e3ee725e23dc\": rpc error: code = NotFound desc = could not find container \"1e2342f5bd2b75c6c6c252ebeda84410590514fd13b61e775db8e3ee725e23dc\": container with ID starting with 1e2342f5bd2b75c6c6c252ebeda84410590514fd13b61e775db8e3ee725e23dc not found: ID does not exist" Oct 30 19:56:39.099982 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.099928 5047 scope.go:122] "RemoveContainer" containerID="4c608ef38e9e55b2ca7e5fcf3d4130b886effe03b01af298b68936f86a6d15dc" Oct 30 19:56:39.114307 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.114290 5047 scope.go:122] "RemoveContainer" containerID="4c608ef38e9e55b2ca7e5fcf3d4130b886effe03b01af298b68936f86a6d15dc" Oct 30 19:56:39.114750 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:56:39.114731 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c608ef38e9e55b2ca7e5fcf3d4130b886effe03b01af298b68936f86a6d15dc\": container with ID starting with 4c608ef38e9e55b2ca7e5fcf3d4130b886effe03b01af298b68936f86a6d15dc not found: ID does not exist" containerID="4c608ef38e9e55b2ca7e5fcf3d4130b886effe03b01af298b68936f86a6d15dc" Oct 30 19:56:39.114821 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.114756 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c608ef38e9e55b2ca7e5fcf3d4130b886effe03b01af298b68936f86a6d15dc"} err="failed to get container status \"4c608ef38e9e55b2ca7e5fcf3d4130b886effe03b01af298b68936f86a6d15dc\": rpc error: code = NotFound desc = could not find container \"4c608ef38e9e55b2ca7e5fcf3d4130b886effe03b01af298b68936f86a6d15dc\": container with ID starting with 4c608ef38e9e55b2ca7e5fcf3d4130b886effe03b01af298b68936f86a6d15dc not found: ID does not exist" Oct 30 19:56:39.137363 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.136992 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r"] Oct 30 19:56:39.141148 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.141127 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-59cc9599d6-6xt6r"] Oct 30 19:56:39.170186 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.170165 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7"] Oct 30 19:56:39.177713 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.177693 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-59f9ed-69cdc58cf-8jfp7"] Oct 30 19:56:39.602883 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:56:39.602849 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:56:39.633170 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.633142 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv"] Oct 30 19:56:39.633966 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.633949 5047 cpu_manager.go:414] "containerMap: removing container" podUID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" containerName="istio-proxy" Oct 30 19:56:39.634064 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.633968 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" containerName="istio-proxy" Oct 30 19:56:39.634064 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.633985 5047 cpu_manager.go:414] "containerMap: removing container" podUID="14168ba7-e407-457a-8a39-b4d0c266135a" containerName="payload-processing" Oct 30 19:56:39.634064 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.633991 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14168ba7-e407-457a-8a39-b4d0c266135a" containerName="payload-processing" Oct 30 19:56:39.634064 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.634039 5047 cpu_manager.go:414] "containerMap: removing container" podUID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" containerName="istio-proxy" Oct 30 19:56:39.634064 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.634046 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" containerName="istio-proxy" Oct 30 19:56:39.634064 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.634056 5047 cpu_manager.go:414] "containerMap: removing container" podUID="d82abce5-4bc3-4a06-aa68-4e688cd1702e" containerName="payload-pre-processing" Oct 30 19:56:39.634064 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.634061 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d82abce5-4bc3-4a06-aa68-4e688cd1702e" containerName="payload-pre-processing" Oct 30 19:56:39.634290 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.634215 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" containerName="istio-proxy" Oct 30 19:56:39.634290 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.634227 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="14168ba7-e407-457a-8a39-b4d0c266135a" containerName="payload-processing" Oct 30 19:56:39.634290 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.634239 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="d82abce5-4bc3-4a06-aa68-4e688cd1702e" containerName="payload-pre-processing" Oct 30 19:56:39.637558 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.637543 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 19:56:39.650363 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.650338 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-315ea2-serving-cert\"" Oct 30 19:56:39.654823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.654801 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv"] Oct 30 19:56:39.670902 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.670871 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a2dff4c-72fb-4569-a49a-f8995370ad75-metrics-certs\") pod \"maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv\" (UID: \"5a2dff4c-72fb-4569-a49a-f8995370ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 19:56:39.671185 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.671165 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a2dff4c-72fb-4569-a49a-f8995370ad75-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv\" (UID: \"5a2dff4c-72fb-4569-a49a-f8995370ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 19:56:39.671269 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.671239 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a2dff4c-72fb-4569-a49a-f8995370ad75-maas-api-tls\") pod \"maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv\" (UID: \"5a2dff4c-72fb-4569-a49a-f8995370ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 19:56:39.671339 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.671320 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6nw2\" (UniqueName: \"kubernetes.io/projected/5a2dff4c-72fb-4569-a49a-f8995370ad75-kube-api-access-v6nw2\") pod \"maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv\" (UID: \"5a2dff4c-72fb-4569-a49a-f8995370ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 19:56:39.772319 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.772290 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a2dff4c-72fb-4569-a49a-f8995370ad75-maas-api-tls\") pod \"maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv\" (UID: \"5a2dff4c-72fb-4569-a49a-f8995370ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 19:56:39.772512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.772365 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6nw2\" (UniqueName: \"kubernetes.io/projected/5a2dff4c-72fb-4569-a49a-f8995370ad75-kube-api-access-v6nw2\") pod \"maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv\" (UID: \"5a2dff4c-72fb-4569-a49a-f8995370ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 19:56:39.772512 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.772420 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a2dff4c-72fb-4569-a49a-f8995370ad75-metrics-certs\") pod \"maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv\" (UID: \"5a2dff4c-72fb-4569-a49a-f8995370ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 19:56:39.772679 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.772658 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a2dff4c-72fb-4569-a49a-f8995370ad75-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv\" (UID: \"5a2dff4c-72fb-4569-a49a-f8995370ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 19:56:39.773526 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.773500 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a2dff4c-72fb-4569-a49a-f8995370ad75-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv\" (UID: \"5a2dff4c-72fb-4569-a49a-f8995370ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 19:56:39.774793 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.774762 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a2dff4c-72fb-4569-a49a-f8995370ad75-maas-api-tls\") pod \"maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv\" (UID: \"5a2dff4c-72fb-4569-a49a-f8995370ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 19:56:39.774895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.774789 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a2dff4c-72fb-4569-a49a-f8995370ad75-metrics-certs\") pod \"maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv\" (UID: \"5a2dff4c-72fb-4569-a49a-f8995370ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 19:56:39.790060 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.790036 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6nw2\" (UniqueName: \"kubernetes.io/projected/5a2dff4c-72fb-4569-a49a-f8995370ad75-kube-api-access-v6nw2\") pod \"maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv\" (UID: \"5a2dff4c-72fb-4569-a49a-f8995370ad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 19:56:39.951664 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:39.951584 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 19:56:40.255046 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:40.254986 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv"] Oct 30 19:56:40.255413 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:56:40.254993 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2dff4c_72fb_4569_a49a_f8995370ad75.slice/crio-2bfdd34645ce932c63f84fb8fb6db5e9f3ada10c51c16436c56a37bb15a9deb1 WatchSource:0}: Error finding container 2bfdd34645ce932c63f84fb8fb6db5e9f3ada10c51c16436c56a37bb15a9deb1: Status 404 returned error can't find the container with id 2bfdd34645ce932c63f84fb8fb6db5e9f3ada10c51c16436c56a37bb15a9deb1 Oct 30 19:56:41.051999 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:41.051918 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14168ba7-e407-457a-8a39-b4d0c266135a" path="/var/lib/kubelet/pods/14168ba7-e407-457a-8a39-b4d0c266135a/volumes" Oct 30 19:56:41.052464 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:41.052449 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d82abce5-4bc3-4a06-aa68-4e688cd1702e" path="/var/lib/kubelet/pods/d82abce5-4bc3-4a06-aa68-4e688cd1702e/volumes" Oct 30 19:56:41.116665 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:41.116625 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" event={"ID":"5a2dff4c-72fb-4569-a49a-f8995370ad75","Type":"ContainerStarted","Data":"09ecea5b6923104c1000d777944b42649f6d5bd5afbd8247aa272a4d712716bc"} Oct 30 19:56:41.116882 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:41.116695 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" event={"ID":"5a2dff4c-72fb-4569-a49a-f8995370ad75","Type":"ContainerStarted","Data":"2bfdd34645ce932c63f84fb8fb6db5e9f3ada10c51c16436c56a37bb15a9deb1"} Oct 30 19:56:41.117042 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:41.116874 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 19:56:41.157580 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:41.157520 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" podStartSLOduration=1.670239605 podStartE2EDuration="2.157500429s" totalImagesPullingTime="487.260824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:41.156398923 +0000 UTC m=+3154.230854281" watchObservedRunningTime="2026-10-30 19:56:41.157500429 +0000 UTC m=+3154.231955788" Oct 30 19:56:43.428734 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:43.428693 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" podUID="3475ab8d-e007-444d-baae-c24d13fd7a9d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:56:43.429237 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:43.429079 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" podUID="14cdb07d-1fb7-4959-baef-21ca0c5b51f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:56:47.150895 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:47.150861 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 19:56:49.113027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:49.112297 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" Oct 30 19:56:49.287729 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:49.287692 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5"] Oct 30 19:56:49.287962 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:49.287922 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" podUID="9a300de1-9957-43a4-a24c-a7186fe1ba64" containerName="istio-proxy" containerID="cri-o://6caa892bb571631653335c8e32f5eca17be821deef4db87a06a6283ec27659b2" gracePeriod=30 Oct 30 19:56:53.428044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:53.427989 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" podUID="3475ab8d-e007-444d-baae-c24d13fd7a9d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:56:53.428044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:53.428037 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" podUID="14cdb07d-1fb7-4959-baef-21ca0c5b51f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:56:54.352787 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.349512 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn"] Oct 30 19:56:54.352787 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.350965 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" containerName="istio-proxy" Oct 30 19:56:54.384237 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.382773 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.403712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.403681 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn"] Oct 30 19:56:54.465036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.461856 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-data\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.465036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.461935 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-workload-socket\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.528027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.518818 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-workload-certs\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.528027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.518968 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-credential-socket\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.528027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.519040 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8856e033-793e-4e57-abbf-ae7ae07cbff4-istiod-ca-cert\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.528027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.519096 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-podinfo\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.528027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.519329 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-token\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.528027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.520903 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsb88\" (UniqueName: \"kubernetes.io/projected/8856e033-793e-4e57-abbf-ae7ae07cbff4-kube-api-access-wsb88\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.528027 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.521019 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-envoy\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.562524 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.533173 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6e6016-openshift-default-dockercfg-slzvs\"" Oct 30 19:56:54.627657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.622834 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-podinfo\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.627657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.622948 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-token\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.627657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.623089 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsb88\" (UniqueName: \"kubernetes.io/projected/8856e033-793e-4e57-abbf-ae7ae07cbff4-kube-api-access-wsb88\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.627657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.623159 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-envoy\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.627657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.623241 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-data\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.627657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.623313 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-workload-socket\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.627657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.623389 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-workload-certs\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.627657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.623454 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-credential-socket\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.627657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.623496 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8856e033-793e-4e57-abbf-ae7ae07cbff4-istiod-ca-cert\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.627657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.624377 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8856e033-793e-4e57-abbf-ae7ae07cbff4-istiod-ca-cert\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.627657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.625790 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-data\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.627657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.626233 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-workload-certs\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.627657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.626769 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-workload-socket\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.627657 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.627165 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-credential-socket\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.634872 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.634820 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-envoy\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.637148 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.637066 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-podinfo\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.647494 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.647464 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-token\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.647494 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.647491 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsb88\" (UniqueName: \"kubernetes.io/projected/8856e033-793e-4e57-abbf-ae7ae07cbff4-kube-api-access-wsb88\") pod \"e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.799859 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.799829 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:56:54.865385 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.865347 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:54.929211 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.929166 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-podinfo\") pod \"9a300de1-9957-43a4-a24c-a7186fe1ba64\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " Oct 30 19:56:54.929355 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.929240 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-data\") pod \"9a300de1-9957-43a4-a24c-a7186fe1ba64\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " Oct 30 19:56:54.929992 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.929884 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-workload-socket\") pod \"9a300de1-9957-43a4-a24c-a7186fe1ba64\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " Oct 30 19:56:54.929992 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.929963 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-credential-socket\") pod \"9a300de1-9957-43a4-a24c-a7186fe1ba64\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " Oct 30 19:56:54.930276 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.930151 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-token\" (UniqueName: \"kubernetes.io/projected/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-token\") pod \"9a300de1-9957-43a4-a24c-a7186fe1ba64\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " Oct 30 19:56:54.930276 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.930189 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a300de1-9957-43a4-a24c-a7186fe1ba64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a300de1-9957-43a4-a24c-a7186fe1ba64-istiod-ca-cert\") pod \"9a300de1-9957-43a4-a24c-a7186fe1ba64\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " Oct 30 19:56:54.930796 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.930485 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-workload-socket" pod "9a300de1-9957-43a4-a24c-a7186fe1ba64" (UID: "9a300de1-9957-43a4-a24c-a7186fe1ba64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:54.930796 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.930668 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-data" pod "9a300de1-9957-43a4-a24c-a7186fe1ba64" (UID: "9a300de1-9957-43a4-a24c-a7186fe1ba64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:54.930796 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.930748 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-credential-socket" pod "9a300de1-9957-43a4-a24c-a7186fe1ba64" (UID: "9a300de1-9957-43a4-a24c-a7186fe1ba64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:54.931503 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.931362 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a300de1-9957-43a4-a24c-a7186fe1ba64-istiod-ca-cert" pod "9a300de1-9957-43a4-a24c-a7186fe1ba64" (UID: "9a300de1-9957-43a4-a24c-a7186fe1ba64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:54.931503 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.931482 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a300de1-9957-43a4-a24c-a7186fe1ba64-kube-api-access-5pdzn\" (UniqueName: \"kubernetes.io/projected/9a300de1-9957-43a4-a24c-a7186fe1ba64-kube-api-access-5pdzn\") pod \"9a300de1-9957-43a4-a24c-a7186fe1ba64\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " Oct 30 19:56:54.931638 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.931552 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-workload-certs\") pod \"9a300de1-9957-43a4-a24c-a7186fe1ba64\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " Oct 30 19:56:54.931638 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.931585 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-envoy\") pod \"9a300de1-9957-43a4-a24c-a7186fe1ba64\" (UID: \"9a300de1-9957-43a4-a24c-a7186fe1ba64\") " Oct 30 19:56:54.932658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.932258 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:54.932658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.932283 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a300de1-9957-43a4-a24c-a7186fe1ba64-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:54.932658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.932298 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:54.932658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.932310 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:54.932658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.932584 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-workload-certs" pod "9a300de1-9957-43a4-a24c-a7186fe1ba64" (UID: "9a300de1-9957-43a4-a24c-a7186fe1ba64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:54.933141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.932729 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-podinfo" pod "9a300de1-9957-43a4-a24c-a7186fe1ba64" (UID: "9a300de1-9957-43a4-a24c-a7186fe1ba64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:56:54.933354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.933326 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-token" pod "9a300de1-9957-43a4-a24c-a7186fe1ba64" (UID: "9a300de1-9957-43a4-a24c-a7186fe1ba64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:54.933964 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.933928 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a300de1-9957-43a4-a24c-a7186fe1ba64-kube-api-access-5pdzn" pod "9a300de1-9957-43a4-a24c-a7186fe1ba64" (UID: "9a300de1-9957-43a4-a24c-a7186fe1ba64"). InnerVolumeSpecName "kube-api-access-5pdzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:54.934849 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:54.934815 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-envoy" pod "9a300de1-9957-43a4-a24c-a7186fe1ba64" (UID: "9a300de1-9957-43a4-a24c-a7186fe1ba64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:55.059130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.034640 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:55.059130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.034680 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pdzn\" (UniqueName: \"kubernetes.io/projected/9a300de1-9957-43a4-a24c-a7186fe1ba64-kube-api-access-5pdzn\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:55.059130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.034695 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:55.059130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.034708 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:55.059130 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.034721 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a300de1-9957-43a4-a24c-a7186fe1ba64-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:55.188694 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:56:55.188626 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8856e033_793e_4e57_abbf_ae7ae07cbff4.slice/crio-ab604bffa1f561377f285b1c1082fff8b1bda62fef8267886e952177e0ebe2fd WatchSource:0}: Error finding container ab604bffa1f561377f285b1c1082fff8b1bda62fef8267886e952177e0ebe2fd: Status 404 returned error can't find the container with id ab604bffa1f561377f285b1c1082fff8b1bda62fef8267886e952177e0ebe2fd Oct 30 19:56:55.189035 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.188990 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn"] Oct 30 19:56:55.192197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.192155 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:56:55.192311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.192246 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:56:55.192311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.192293 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:56:55.413060 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.411290 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" event={"ID":"8856e033-793e-4e57-abbf-ae7ae07cbff4","Type":"ContainerStarted","Data":"b181b06d3a5acaf3e8e45a05e781d6dc9fb86f6b8931d5432014a39e792bd0ff"} Oct 30 19:56:55.413060 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.411370 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" event={"ID":"8856e033-793e-4e57-abbf-ae7ae07cbff4","Type":"ContainerStarted","Data":"ab604bffa1f561377f285b1c1082fff8b1bda62fef8267886e952177e0ebe2fd"} Oct 30 19:56:55.413536 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.413508 5047 generic.go:356] "Generic (PLEG): container finished" podID="9a300de1-9957-43a4-a24c-a7186fe1ba64" containerID="6caa892bb571631653335c8e32f5eca17be821deef4db87a06a6283ec27659b2" exitCode=0 Oct 30 19:56:55.413597 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.413578 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" event={"ID":"9a300de1-9957-43a4-a24c-a7186fe1ba64","Type":"ContainerDied","Data":"6caa892bb571631653335c8e32f5eca17be821deef4db87a06a6283ec27659b2"} Oct 30 19:56:55.413658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.413620 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" event={"ID":"9a300de1-9957-43a4-a24c-a7186fe1ba64","Type":"ContainerDied","Data":"f70cbf2bde7642b95e5803b60e2a25d52c0997d369e17bad759c0e4407d154d5"} Oct 30 19:56:55.413658 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.413642 5047 scope.go:122] "RemoveContainer" containerID="6caa892bb571631653335c8e32f5eca17be821deef4db87a06a6283ec27659b2" Oct 30 19:56:55.413766 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.413754 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5" Oct 30 19:56:55.436816 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.436589 5047 scope.go:122] "RemoveContainer" containerID="196eef2e0a36f718f9ad15e2d0c83509829c6f11ed1c90ce834239f8fbfb797f" Oct 30 19:56:55.448209 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.448188 5047 scope.go:122] "RemoveContainer" containerID="6caa892bb571631653335c8e32f5eca17be821deef4db87a06a6283ec27659b2" Oct 30 19:56:55.448536 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:56:55.448514 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6caa892bb571631653335c8e32f5eca17be821deef4db87a06a6283ec27659b2\": container with ID starting with 6caa892bb571631653335c8e32f5eca17be821deef4db87a06a6283ec27659b2 not found: ID does not exist" containerID="6caa892bb571631653335c8e32f5eca17be821deef4db87a06a6283ec27659b2" Oct 30 19:56:55.448618 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.448547 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6caa892bb571631653335c8e32f5eca17be821deef4db87a06a6283ec27659b2"} err="failed to get container status \"6caa892bb571631653335c8e32f5eca17be821deef4db87a06a6283ec27659b2\": rpc error: code = NotFound desc = could not find container \"6caa892bb571631653335c8e32f5eca17be821deef4db87a06a6283ec27659b2\": container with ID starting with 6caa892bb571631653335c8e32f5eca17be821deef4db87a06a6283ec27659b2 not found: ID does not exist" Oct 30 19:56:55.448618 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.448575 5047 scope.go:122] "RemoveContainer" containerID="196eef2e0a36f718f9ad15e2d0c83509829c6f11ed1c90ce834239f8fbfb797f" Oct 30 19:56:55.448891 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:56:55.448874 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196eef2e0a36f718f9ad15e2d0c83509829c6f11ed1c90ce834239f8fbfb797f\": container with ID starting with 196eef2e0a36f718f9ad15e2d0c83509829c6f11ed1c90ce834239f8fbfb797f not found: ID does not exist" containerID="196eef2e0a36f718f9ad15e2d0c83509829c6f11ed1c90ce834239f8fbfb797f" Oct 30 19:56:55.448950 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.448896 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196eef2e0a36f718f9ad15e2d0c83509829c6f11ed1c90ce834239f8fbfb797f"} err="failed to get container status \"196eef2e0a36f718f9ad15e2d0c83509829c6f11ed1c90ce834239f8fbfb797f\": rpc error: code = NotFound desc = could not find container \"196eef2e0a36f718f9ad15e2d0c83509829c6f11ed1c90ce834239f8fbfb797f\": container with ID starting with 196eef2e0a36f718f9ad15e2d0c83509829c6f11ed1c90ce834239f8fbfb797f not found: ID does not exist" Oct 30 19:56:55.453223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.453170 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" podStartSLOduration=1.453153597 podStartE2EDuration="1.453153597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:55.450467574 +0000 UTC m=+3168.524922955" watchObservedRunningTime="2026-10-30 19:56:55.453153597 +0000 UTC m=+3168.527608956" Oct 30 19:56:55.485322 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.485280 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5"] Oct 30 19:56:55.489553 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.489520 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5250ee-openshift-default-d7d95dcdc-hwsl5"] Oct 30 19:56:55.865879 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:55.865845 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:56.651047 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:56.651019 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" Oct 30 19:56:56.768032 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:56.764431 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14cdb07d-1fb7-4959-baef-21ca0c5b51f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cdb07d-1fb7-4959-baef-21ca0c5b51f1-plugins-config-volume\") pod \"14cdb07d-1fb7-4959-baef-21ca0c5b51f1\" (UID: \"14cdb07d-1fb7-4959-baef-21ca0c5b51f1\") " Oct 30 19:56:56.777139 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:56.774468 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cdb07d-1fb7-4959-baef-21ca0c5b51f1-plugins-config-volume" pod "14cdb07d-1fb7-4959-baef-21ca0c5b51f1" (UID: "14cdb07d-1fb7-4959-baef-21ca0c5b51f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:56.777139 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:56.774974 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14cdb07d-1fb7-4959-baef-21ca0c5b51f1-kube-api-access-h6zdb\" (UniqueName: \"kubernetes.io/projected/14cdb07d-1fb7-4959-baef-21ca0c5b51f1-kube-api-access-h6zdb\") pod \"14cdb07d-1fb7-4959-baef-21ca0c5b51f1\" (UID: \"14cdb07d-1fb7-4959-baef-21ca0c5b51f1\") " Oct 30 19:56:56.779210 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:56.778060 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cdb07d-1fb7-4959-baef-21ca0c5b51f1-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:56.781222 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:56.781177 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14cdb07d-1fb7-4959-baef-21ca0c5b51f1-kube-api-access-h6zdb" pod "14cdb07d-1fb7-4959-baef-21ca0c5b51f1" (UID: "14cdb07d-1fb7-4959-baef-21ca0c5b51f1"). InnerVolumeSpecName "kube-api-access-h6zdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:56.821367 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:56.821343 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" Oct 30 19:56:56.874592 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:56.874560 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:56.878693 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:56.878634 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3475ab8d-e007-444d-baae-c24d13fd7a9d-kube-api-access-t9xr4\" (UniqueName: \"kubernetes.io/projected/3475ab8d-e007-444d-baae-c24d13fd7a9d-kube-api-access-t9xr4\") pod \"3475ab8d-e007-444d-baae-c24d13fd7a9d\" (UID: \"3475ab8d-e007-444d-baae-c24d13fd7a9d\") " Oct 30 19:56:56.879054 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:56.878996 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3475ab8d-e007-444d-baae-c24d13fd7a9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3475ab8d-e007-444d-baae-c24d13fd7a9d-plugins-config-volume\") pod \"3475ab8d-e007-444d-baae-c24d13fd7a9d\" (UID: \"3475ab8d-e007-444d-baae-c24d13fd7a9d\") " Oct 30 19:56:56.879388 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:56.879367 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3475ab8d-e007-444d-baae-c24d13fd7a9d-plugins-config-volume" pod "3475ab8d-e007-444d-baae-c24d13fd7a9d" (UID: "3475ab8d-e007-444d-baae-c24d13fd7a9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:56.879628 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:56.879606 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6zdb\" (UniqueName: \"kubernetes.io/projected/14cdb07d-1fb7-4959-baef-21ca0c5b51f1-kube-api-access-h6zdb\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:56.879731 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:56.879631 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3475ab8d-e007-444d-baae-c24d13fd7a9d-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:56.881321 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:56.881301 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3475ab8d-e007-444d-baae-c24d13fd7a9d-kube-api-access-t9xr4" pod "3475ab8d-e007-444d-baae-c24d13fd7a9d" (UID: "3475ab8d-e007-444d-baae-c24d13fd7a9d"). InnerVolumeSpecName "kube-api-access-t9xr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:56.980968 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:56.980925 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9xr4\" (UniqueName: \"kubernetes.io/projected/3475ab8d-e007-444d-baae-c24d13fd7a9d-kube-api-access-t9xr4\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:56:57.051834 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.051759 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a300de1-9957-43a4-a24c-a7186fe1ba64" path="/var/lib/kubelet/pods/9a300de1-9957-43a4-a24c-a7186fe1ba64/volumes" Oct 30 19:56:57.433204 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.433166 5047 generic.go:356] "Generic (PLEG): container finished" podID="14cdb07d-1fb7-4959-baef-21ca0c5b51f1" containerID="b5c30e03408f00e6322ef7454cc134976d667ebb4c330bc53073076ad4bdb91d" exitCode=0 Oct 30 19:56:57.433416 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.433211 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" Oct 30 19:56:57.433416 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.433247 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" event={"ID":"14cdb07d-1fb7-4959-baef-21ca0c5b51f1","Type":"ContainerDied","Data":"b5c30e03408f00e6322ef7454cc134976d667ebb4c330bc53073076ad4bdb91d"} Oct 30 19:56:57.433416 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.433300 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d" event={"ID":"14cdb07d-1fb7-4959-baef-21ca0c5b51f1","Type":"ContainerDied","Data":"b9bdda9317a763f044e918127059e83d328cde81f3e9a48505d65951655897ec"} Oct 30 19:56:57.433416 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.433324 5047 scope.go:122] "RemoveContainer" containerID="b5c30e03408f00e6322ef7454cc134976d667ebb4c330bc53073076ad4bdb91d" Oct 30 19:56:57.434823 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.434802 5047 generic.go:356] "Generic (PLEG): container finished" podID="3475ab8d-e007-444d-baae-c24d13fd7a9d" containerID="7b3c74a68c63632d0dd97f1d32d4b2a8e10a66262f4244bd5971dd33db678edb" exitCode=0 Oct 30 19:56:57.434959 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.434836 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" Oct 30 19:56:57.434959 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.434907 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" event={"ID":"3475ab8d-e007-444d-baae-c24d13fd7a9d","Type":"ContainerDied","Data":"7b3c74a68c63632d0dd97f1d32d4b2a8e10a66262f4244bd5971dd33db678edb"} Oct 30 19:56:57.435112 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.434969 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc" event={"ID":"3475ab8d-e007-444d-baae-c24d13fd7a9d","Type":"ContainerDied","Data":"2534674038d467157fec7ae71dc2ae23fd9fca509e923f064b9da641d0785481"} Oct 30 19:56:57.435424 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.435353 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:57.437677 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.437654 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:56:57.446416 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.446395 5047 scope.go:122] "RemoveContainer" containerID="b5c30e03408f00e6322ef7454cc134976d667ebb4c330bc53073076ad4bdb91d" Oct 30 19:56:57.446749 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:56:57.446729 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c30e03408f00e6322ef7454cc134976d667ebb4c330bc53073076ad4bdb91d\": container with ID starting with b5c30e03408f00e6322ef7454cc134976d667ebb4c330bc53073076ad4bdb91d not found: ID does not exist" containerID="b5c30e03408f00e6322ef7454cc134976d667ebb4c330bc53073076ad4bdb91d" Oct 30 19:56:57.446832 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.446758 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c30e03408f00e6322ef7454cc134976d667ebb4c330bc53073076ad4bdb91d"} err="failed to get container status \"b5c30e03408f00e6322ef7454cc134976d667ebb4c330bc53073076ad4bdb91d\": rpc error: code = NotFound desc = could not find container \"b5c30e03408f00e6322ef7454cc134976d667ebb4c330bc53073076ad4bdb91d\": container with ID starting with b5c30e03408f00e6322ef7454cc134976d667ebb4c330bc53073076ad4bdb91d not found: ID does not exist" Oct 30 19:56:57.446832 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.446775 5047 scope.go:122] "RemoveContainer" containerID="7b3c74a68c63632d0dd97f1d32d4b2a8e10a66262f4244bd5971dd33db678edb" Oct 30 19:56:57.463699 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.463565 5047 scope.go:122] "RemoveContainer" containerID="7b3c74a68c63632d0dd97f1d32d4b2a8e10a66262f4244bd5971dd33db678edb" Oct 30 19:56:57.463944 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:56:57.463917 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3c74a68c63632d0dd97f1d32d4b2a8e10a66262f4244bd5971dd33db678edb\": container with ID starting with 7b3c74a68c63632d0dd97f1d32d4b2a8e10a66262f4244bd5971dd33db678edb not found: ID does not exist" containerID="7b3c74a68c63632d0dd97f1d32d4b2a8e10a66262f4244bd5971dd33db678edb" Oct 30 19:56:57.464059 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.463955 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3c74a68c63632d0dd97f1d32d4b2a8e10a66262f4244bd5971dd33db678edb"} err="failed to get container status \"7b3c74a68c63632d0dd97f1d32d4b2a8e10a66262f4244bd5971dd33db678edb\": rpc error: code = NotFound desc = could not find container \"7b3c74a68c63632d0dd97f1d32d4b2a8e10a66262f4244bd5971dd33db678edb\": container with ID starting with 7b3c74a68c63632d0dd97f1d32d4b2a8e10a66262f4244bd5971dd33db678edb not found: ID does not exist" Oct 30 19:56:57.480238 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.480212 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc"] Oct 30 19:56:57.485529 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.485496 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5250ee-7876984d4c-9z2pc"] Oct 30 19:56:57.514472 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.514445 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d"] Oct 30 19:56:57.518234 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:57.518202 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-5250ee-854db4f576-xzp8d"] Oct 30 19:56:59.051274 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:59.051242 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14cdb07d-1fb7-4959-baef-21ca0c5b51f1" path="/var/lib/kubelet/pods/14cdb07d-1fb7-4959-baef-21ca0c5b51f1/volumes" Oct 30 19:56:59.051772 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:59.051755 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3475ab8d-e007-444d-baae-c24d13fd7a9d" path="/var/lib/kubelet/pods/3475ab8d-e007-444d-baae-c24d13fd7a9d/volumes" Oct 30 19:56:59.961226 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:56:59.961186 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w"] Oct 30 19:57:00.002198 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.001895 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4"] Oct 30 19:57:00.192456 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.192414 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff"] Oct 30 19:57:00.194040 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.193984 5047 cpu_manager.go:414] "containerMap: removing container" podUID="9a300de1-9957-43a4-a24c-a7186fe1ba64" containerName="istio-proxy" Oct 30 19:57:00.194040 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.194027 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a300de1-9957-43a4-a24c-a7186fe1ba64" containerName="istio-proxy" Oct 30 19:57:00.195495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.194092 5047 cpu_manager.go:414] "containerMap: removing container" podUID="14cdb07d-1fb7-4959-baef-21ca0c5b51f1" containerName="payload-processing" Oct 30 19:57:00.195495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.194101 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cdb07d-1fb7-4959-baef-21ca0c5b51f1" containerName="payload-processing" Oct 30 19:57:00.195495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.194131 5047 cpu_manager.go:414] "containerMap: removing container" podUID="3475ab8d-e007-444d-baae-c24d13fd7a9d" containerName="payload-pre-processing" Oct 30 19:57:00.195495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.194139 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3475ab8d-e007-444d-baae-c24d13fd7a9d" containerName="payload-pre-processing" Oct 30 19:57:00.195495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.194327 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a300de1-9957-43a4-a24c-a7186fe1ba64" containerName="istio-proxy" Oct 30 19:57:00.195495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.194348 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cdb07d-1fb7-4959-baef-21ca0c5b51f1" containerName="payload-processing" Oct 30 19:57:00.195495 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.194362 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="3475ab8d-e007-444d-baae-c24d13fd7a9d" containerName="payload-pre-processing" Oct 30 19:57:00.209826 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.200814 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" Oct 30 19:57:00.235148 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.223498 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2afc11\"" Oct 30 19:57:00.235148 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.223633 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2afc11-dockercfg-5dkjs\"" Oct 30 19:57:00.310040 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.304042 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvjzb\" (UniqueName: \"kubernetes.io/projected/0c851817-5127-4d01-91db-5f6bf9e8a062-kube-api-access-mvjzb\") pod \"payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff\" (UID: \"0c851817-5127-4d01-91db-5f6bf9e8a062\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" Oct 30 19:57:00.310040 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.304102 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c851817-5127-4d01-91db-5f6bf9e8a062-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff\" (UID: \"0c851817-5127-4d01-91db-5f6bf9e8a062\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" Oct 30 19:57:00.425216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.414998 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvjzb\" (UniqueName: \"kubernetes.io/projected/0c851817-5127-4d01-91db-5f6bf9e8a062-kube-api-access-mvjzb\") pod \"payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff\" (UID: \"0c851817-5127-4d01-91db-5f6bf9e8a062\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" Oct 30 19:57:00.425216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.415204 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c851817-5127-4d01-91db-5f6bf9e8a062-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff\" (UID: \"0c851817-5127-4d01-91db-5f6bf9e8a062\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" Oct 30 19:57:00.425216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.416862 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c851817-5127-4d01-91db-5f6bf9e8a062-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff\" (UID: \"0c851817-5127-4d01-91db-5f6bf9e8a062\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" Oct 30 19:57:00.425216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.422923 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff"] Oct 30 19:57:00.443034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.439837 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvjzb\" (UniqueName: \"kubernetes.io/projected/0c851817-5127-4d01-91db-5f6bf9e8a062-kube-api-access-mvjzb\") pod \"payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff\" (UID: \"0c851817-5127-4d01-91db-5f6bf9e8a062\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" Oct 30 19:57:00.715026 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:00.714408 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" Oct 30 19:57:01.119511 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:57:01.119475 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c851817_5127_4d01_91db_5f6bf9e8a062.slice/crio-12333c69576d536e43723d79ce7cf7ae0e13b3af497a7da1f157cc9c19cf3c5a WatchSource:0}: Error finding container 12333c69576d536e43723d79ce7cf7ae0e13b3af497a7da1f157cc9c19cf3c5a: Status 404 returned error can't find the container with id 12333c69576d536e43723d79ce7cf7ae0e13b3af497a7da1f157cc9c19cf3c5a Oct 30 19:57:01.119636 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:01.119552 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff"] Oct 30 19:57:01.477445 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:01.477352 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" event={"ID":"0c851817-5127-4d01-91db-5f6bf9e8a062","Type":"ContainerStarted","Data":"da31b0f320d251bf4d850a8d683602d67c122471a95cea3e4c685b499381a20a"} Oct 30 19:57:01.477445 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:01.477417 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" event={"ID":"0c851817-5127-4d01-91db-5f6bf9e8a062","Type":"ContainerStarted","Data":"12333c69576d536e43723d79ce7cf7ae0e13b3af497a7da1f157cc9c19cf3c5a"} Oct 30 19:57:01.477955 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:01.477485 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" Oct 30 19:57:01.529546 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:01.528354 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" podStartSLOduration=1.528329296 podStartE2EDuration="1.528329296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:01.514355932 +0000 UTC m=+3174.588811292" watchObservedRunningTime="2026-10-30 19:57:01.528329296 +0000 UTC m=+3174.602784656" Oct 30 19:57:04.962202 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:04.962129 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" podUID="c8d7d671-e339-41da-94ff-0c4b553a4147" containerName="payload-processing" containerID="cri-o://a17cd27f75b0fba5a6e993271e547ad469a6dcee8dbd1a43096e5b01a79a39d5" gracePeriod=25 Oct 30 19:57:05.003113 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:05.003048 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" podUID="87185d96-e36e-47f7-8ea0-dd2e411b01ac" containerName="payload-pre-processing" containerID="cri-o://cad4e96051190a8ce5801daaa629efb25eea155d8bfa93431a058a2079f5891b" gracePeriod=25 Oct 30 19:57:07.077231 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:07.077182 5047 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","poddec5c9c7-1994-4064-b2fe-ef9f288c53ba"] err="unable to destroy cgroup paths for cgroup [kubepods burstable poddec5c9c7-1994-4064-b2fe-ef9f288c53ba] : Timed out while waiting for systemd to remove kubepods-burstable-poddec5c9c7_1994_4064_b2fe_ef9f288c53ba.slice" Oct 30 19:57:07.077231 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:57:07.077237 5047 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable poddec5c9c7-1994-4064-b2fe-ef9f288c53ba] : unable to destroy cgroup paths for cgroup [kubepods burstable poddec5c9c7-1994-4064-b2fe-ef9f288c53ba] : Timed out while waiting for systemd to remove kubepods-burstable-poddec5c9c7_1994_4064_b2fe_ef9f288c53ba.slice" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" podUID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" Oct 30 19:57:07.527409 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:07.527374 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf" Oct 30 19:57:07.587549 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:07.587510 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf"] Oct 30 19:57:07.596309 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:07.596276 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f44f74-openshift-default-677cb5496f-7flvf"] Oct 30 19:57:09.072136 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:09.072093 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec5c9c7-1994-4064-b2fe-ef9f288c53ba" path="/var/lib/kubelet/pods/dec5c9c7-1994-4064-b2fe-ef9f288c53ba/volumes" Oct 30 19:57:09.569245 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:09.569206 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" podUID="87185d96-e36e-47f7-8ea0-dd2e411b01ac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:57:09.602530 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:57:09.602496 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:57:10.806100 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:10.806056 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" podUID="c8d7d671-e339-41da-94ff-0c4b553a4147" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:57:10.836097 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:10.835970 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj"] Oct 30 19:57:10.841128 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:10.841040 5047 cpu_manager.go:414] "containerMap: removing container" podUID="9a300de1-9957-43a4-a24c-a7186fe1ba64" containerName="istio-proxy" Oct 30 19:57:10.841385 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:10.841365 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a300de1-9957-43a4-a24c-a7186fe1ba64" containerName="istio-proxy" Oct 30 19:57:10.841777 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:10.841759 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a300de1-9957-43a4-a24c-a7186fe1ba64" containerName="istio-proxy" Oct 30 19:57:10.865140 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:10.859905 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" Oct 30 19:57:10.865835 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:10.865781 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj"] Oct 30 19:57:10.877434 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:10.874283 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-59f9ed\"" Oct 30 19:57:10.887088 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:10.887055 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-59f9ed-dockercfg-r7npr\"" Oct 30 19:57:10.997400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:10.997361 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7kgq\" (UniqueName: \"kubernetes.io/projected/f9a562ca-d97d-4470-8f56-38ce5ae4b9a4-kube-api-access-t7kgq\") pod \"payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj\" (UID: \"f9a562ca-d97d-4470-8f56-38ce5ae4b9a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" Oct 30 19:57:10.997596 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:10.997535 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a562ca-d97d-4470-8f56-38ce5ae4b9a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj\" (UID: \"f9a562ca-d97d-4470-8f56-38ce5ae4b9a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" Oct 30 19:57:11.099310 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:11.099280 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a562ca-d97d-4470-8f56-38ce5ae4b9a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj\" (UID: \"f9a562ca-d97d-4470-8f56-38ce5ae4b9a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" Oct 30 19:57:11.099559 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:11.099542 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7kgq\" (UniqueName: \"kubernetes.io/projected/f9a562ca-d97d-4470-8f56-38ce5ae4b9a4-kube-api-access-t7kgq\") pod \"payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj\" (UID: \"f9a562ca-d97d-4470-8f56-38ce5ae4b9a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" Oct 30 19:57:11.099938 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:11.099918 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a562ca-d97d-4470-8f56-38ce5ae4b9a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj\" (UID: \"f9a562ca-d97d-4470-8f56-38ce5ae4b9a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" Oct 30 19:57:11.116401 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:11.116373 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7kgq\" (UniqueName: \"kubernetes.io/projected/f9a562ca-d97d-4470-8f56-38ce5ae4b9a4-kube-api-access-t7kgq\") pod \"payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj\" (UID: \"f9a562ca-d97d-4470-8f56-38ce5ae4b9a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" Oct 30 19:57:11.196050 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:11.195985 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" Oct 30 19:57:11.505880 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:57:11.505837 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a562ca_d97d_4470_8f56_38ce5ae4b9a4.slice/crio-f4bf9264590d6745433c0e74f9e324225351c3b3afda884d1d22b7e52ff24908 WatchSource:0}: Error finding container f4bf9264590d6745433c0e74f9e324225351c3b3afda884d1d22b7e52ff24908: Status 404 returned error can't find the container with id f4bf9264590d6745433c0e74f9e324225351c3b3afda884d1d22b7e52ff24908 Oct 30 19:57:11.506710 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:11.506680 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj"] Oct 30 19:57:11.814639 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:11.814552 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" event={"ID":"f9a562ca-d97d-4470-8f56-38ce5ae4b9a4","Type":"ContainerStarted","Data":"0eee5bd4ad4b20bf306f4dce595e7be0ebbb90ad0ef1ad4ce4f0801f7d873f25"} Oct 30 19:57:11.814639 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:11.814605 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" event={"ID":"f9a562ca-d97d-4470-8f56-38ce5ae4b9a4","Type":"ContainerStarted","Data":"f4bf9264590d6745433c0e74f9e324225351c3b3afda884d1d22b7e52ff24908"} Oct 30 19:57:11.814639 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:11.814627 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" Oct 30 19:57:11.855337 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:11.855281 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" podStartSLOduration=1.855267264 podStartE2EDuration="1.855267264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:11.853117567 +0000 UTC m=+3184.927572927" watchObservedRunningTime="2026-10-30 19:57:11.855267264 +0000 UTC m=+3184.929722622" Oct 30 19:57:12.541028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:12.534149 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" Oct 30 19:57:19.569226 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:19.569134 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" podUID="87185d96-e36e-47f7-8ea0-dd2e411b01ac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:57:20.577979 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:20.577934 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" podUID="c8d7d671-e339-41da-94ff-0c4b553a4147" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:57:22.824268 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:22.824230 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" Oct 30 19:57:25.409983 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.409949 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" Oct 30 19:57:25.457888 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.457860 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" Oct 30 19:57:25.532074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.525744 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87185d96-e36e-47f7-8ea0-dd2e411b01ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87185d96-e36e-47f7-8ea0-dd2e411b01ac-plugins-config-volume\") pod \"87185d96-e36e-47f7-8ea0-dd2e411b01ac\" (UID: \"87185d96-e36e-47f7-8ea0-dd2e411b01ac\") " Oct 30 19:57:25.532074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.525800 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8d7d671-e339-41da-94ff-0c4b553a4147-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d7d671-e339-41da-94ff-0c4b553a4147-plugins-config-volume\") pod \"c8d7d671-e339-41da-94ff-0c4b553a4147\" (UID: \"c8d7d671-e339-41da-94ff-0c4b553a4147\") " Oct 30 19:57:25.532074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.525850 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d7d671-e339-41da-94ff-0c4b553a4147-kube-api-access-gb4cm\" (UniqueName: \"kubernetes.io/projected/c8d7d671-e339-41da-94ff-0c4b553a4147-kube-api-access-gb4cm\") pod \"c8d7d671-e339-41da-94ff-0c4b553a4147\" (UID: \"c8d7d671-e339-41da-94ff-0c4b553a4147\") " Oct 30 19:57:25.532074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.531554 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87185d96-e36e-47f7-8ea0-dd2e411b01ac-plugins-config-volume" pod "87185d96-e36e-47f7-8ea0-dd2e411b01ac" (UID: "87185d96-e36e-47f7-8ea0-dd2e411b01ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:25.532432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.532159 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d7d671-e339-41da-94ff-0c4b553a4147-plugins-config-volume" pod "c8d7d671-e339-41da-94ff-0c4b553a4147" (UID: "c8d7d671-e339-41da-94ff-0c4b553a4147"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:25.534967 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.534697 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87185d96-e36e-47f7-8ea0-dd2e411b01ac-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:57:25.534967 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.534730 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d7d671-e339-41da-94ff-0c4b553a4147-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:57:25.535247 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.535098 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d7d671-e339-41da-94ff-0c4b553a4147-kube-api-access-gb4cm" pod "c8d7d671-e339-41da-94ff-0c4b553a4147" (UID: "c8d7d671-e339-41da-94ff-0c4b553a4147"). InnerVolumeSpecName "kube-api-access-gb4cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:25.635784 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.635748 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87185d96-e36e-47f7-8ea0-dd2e411b01ac-kube-api-access-ln2rq\" (UniqueName: \"kubernetes.io/projected/87185d96-e36e-47f7-8ea0-dd2e411b01ac-kube-api-access-ln2rq\") pod \"87185d96-e36e-47f7-8ea0-dd2e411b01ac\" (UID: \"87185d96-e36e-47f7-8ea0-dd2e411b01ac\") " Oct 30 19:57:25.636353 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.636330 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb4cm\" (UniqueName: \"kubernetes.io/projected/c8d7d671-e339-41da-94ff-0c4b553a4147-kube-api-access-gb4cm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:57:25.637975 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.637949 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87185d96-e36e-47f7-8ea0-dd2e411b01ac-kube-api-access-ln2rq" pod "87185d96-e36e-47f7-8ea0-dd2e411b01ac" (UID: "87185d96-e36e-47f7-8ea0-dd2e411b01ac"). InnerVolumeSpecName "kube-api-access-ln2rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:25.737793 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.737748 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln2rq\" (UniqueName: \"kubernetes.io/projected/87185d96-e36e-47f7-8ea0-dd2e411b01ac-kube-api-access-ln2rq\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:57:25.977215 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.976667 5047 generic.go:356] "Generic (PLEG): container finished" podID="c8d7d671-e339-41da-94ff-0c4b553a4147" containerID="a17cd27f75b0fba5a6e993271e547ad469a6dcee8dbd1a43096e5b01a79a39d5" exitCode=0 Oct 30 19:57:25.977215 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.976941 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" event={"ID":"c8d7d671-e339-41da-94ff-0c4b553a4147","Type":"ContainerDied","Data":"a17cd27f75b0fba5a6e993271e547ad469a6dcee8dbd1a43096e5b01a79a39d5"} Oct 30 19:57:25.977215 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.977030 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" event={"ID":"c8d7d671-e339-41da-94ff-0c4b553a4147","Type":"ContainerDied","Data":"33e5d7be6ca16f9bd67d231a0f2d727c26f1f35bced5213c8194800fffd5ef81"} Oct 30 19:57:25.977215 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.977066 5047 scope.go:122] "RemoveContainer" containerID="a17cd27f75b0fba5a6e993271e547ad469a6dcee8dbd1a43096e5b01a79a39d5" Oct 30 19:57:25.977215 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.977164 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w" Oct 30 19:57:25.982072 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.981847 5047 generic.go:356] "Generic (PLEG): container finished" podID="87185d96-e36e-47f7-8ea0-dd2e411b01ac" containerID="cad4e96051190a8ce5801daaa629efb25eea155d8bfa93431a058a2079f5891b" exitCode=0 Oct 30 19:57:25.986030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.982348 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" event={"ID":"87185d96-e36e-47f7-8ea0-dd2e411b01ac","Type":"ContainerDied","Data":"cad4e96051190a8ce5801daaa629efb25eea155d8bfa93431a058a2079f5891b"} Oct 30 19:57:25.986030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.982406 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" event={"ID":"87185d96-e36e-47f7-8ea0-dd2e411b01ac","Type":"ContainerDied","Data":"f38c9c69847c23d6c62c5086d56a68a88a9939d90dd371fc5dd0a8f1a67d62f2"} Oct 30 19:57:25.986030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:25.982488 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4" Oct 30 19:57:26.007716 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:26.007693 5047 scope.go:122] "RemoveContainer" containerID="a17cd27f75b0fba5a6e993271e547ad469a6dcee8dbd1a43096e5b01a79a39d5" Oct 30 19:57:26.008117 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:57:26.008092 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17cd27f75b0fba5a6e993271e547ad469a6dcee8dbd1a43096e5b01a79a39d5\": container with ID starting with a17cd27f75b0fba5a6e993271e547ad469a6dcee8dbd1a43096e5b01a79a39d5 not found: ID does not exist" containerID="a17cd27f75b0fba5a6e993271e547ad469a6dcee8dbd1a43096e5b01a79a39d5" Oct 30 19:57:26.008223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:26.008126 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17cd27f75b0fba5a6e993271e547ad469a6dcee8dbd1a43096e5b01a79a39d5"} err="failed to get container status \"a17cd27f75b0fba5a6e993271e547ad469a6dcee8dbd1a43096e5b01a79a39d5\": rpc error: code = NotFound desc = could not find container \"a17cd27f75b0fba5a6e993271e547ad469a6dcee8dbd1a43096e5b01a79a39d5\": container with ID starting with a17cd27f75b0fba5a6e993271e547ad469a6dcee8dbd1a43096e5b01a79a39d5 not found: ID does not exist" Oct 30 19:57:26.008223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:26.008145 5047 scope.go:122] "RemoveContainer" containerID="cad4e96051190a8ce5801daaa629efb25eea155d8bfa93431a058a2079f5891b" Oct 30 19:57:26.018864 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:26.018843 5047 scope.go:122] "RemoveContainer" containerID="cad4e96051190a8ce5801daaa629efb25eea155d8bfa93431a058a2079f5891b" Oct 30 19:57:26.019180 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:57:26.019157 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad4e96051190a8ce5801daaa629efb25eea155d8bfa93431a058a2079f5891b\": container with ID starting with cad4e96051190a8ce5801daaa629efb25eea155d8bfa93431a058a2079f5891b not found: ID does not exist" containerID="cad4e96051190a8ce5801daaa629efb25eea155d8bfa93431a058a2079f5891b" Oct 30 19:57:26.019239 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:26.019186 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad4e96051190a8ce5801daaa629efb25eea155d8bfa93431a058a2079f5891b"} err="failed to get container status \"cad4e96051190a8ce5801daaa629efb25eea155d8bfa93431a058a2079f5891b\": rpc error: code = NotFound desc = could not find container \"cad4e96051190a8ce5801daaa629efb25eea155d8bfa93431a058a2079f5891b\": container with ID starting with cad4e96051190a8ce5801daaa629efb25eea155d8bfa93431a058a2079f5891b not found: ID does not exist" Oct 30 19:57:26.043664 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:26.043639 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w"] Oct 30 19:57:26.049246 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:26.049224 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-aae4d8-859d677588-p9n4w"] Oct 30 19:57:26.079526 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:26.079498 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4"] Oct 30 19:57:26.083590 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:26.083570 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-aae4d8-6f6d78bc8c-smvt4"] Oct 30 19:57:27.050919 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:27.050884 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87185d96-e36e-47f7-8ea0-dd2e411b01ac" path="/var/lib/kubelet/pods/87185d96-e36e-47f7-8ea0-dd2e411b01ac/volumes" Oct 30 19:57:27.051508 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:27.051487 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8d7d671-e339-41da-94ff-0c4b553a4147" path="/var/lib/kubelet/pods/c8d7d671-e339-41da-94ff-0c4b553a4147/volumes" Oct 30 19:57:34.125564 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:34.125523 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc"] Oct 30 19:57:39.126356 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:39.126314 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" podUID="b867838d-f9a0-45e6-abd3-1d89dabbe6ca" containerName="payload-processing" containerID="cri-o://12929bf02542ecce769c607d3254b4040402d54e6832cb6281e1fa73d7f090c3" gracePeriod=25 Oct 30 19:57:39.420871 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:39.420842 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" Oct 30 19:57:39.521796 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:39.521763 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b867838d-f9a0-45e6-abd3-1d89dabbe6ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b867838d-f9a0-45e6-abd3-1d89dabbe6ca-plugins-config-volume\") pod \"b867838d-f9a0-45e6-abd3-1d89dabbe6ca\" (UID: \"b867838d-f9a0-45e6-abd3-1d89dabbe6ca\") " Oct 30 19:57:39.522169 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:39.522147 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b867838d-f9a0-45e6-abd3-1d89dabbe6ca-kube-api-access-bmm86\" (UniqueName: \"kubernetes.io/projected/b867838d-f9a0-45e6-abd3-1d89dabbe6ca-kube-api-access-bmm86\") pod \"b867838d-f9a0-45e6-abd3-1d89dabbe6ca\" (UID: \"b867838d-f9a0-45e6-abd3-1d89dabbe6ca\") " Oct 30 19:57:39.522253 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:39.522199 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b867838d-f9a0-45e6-abd3-1d89dabbe6ca-plugins-config-volume" pod "b867838d-f9a0-45e6-abd3-1d89dabbe6ca" (UID: "b867838d-f9a0-45e6-abd3-1d89dabbe6ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:39.522701 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:39.522677 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b867838d-f9a0-45e6-abd3-1d89dabbe6ca-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:57:39.524425 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:39.524394 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b867838d-f9a0-45e6-abd3-1d89dabbe6ca-kube-api-access-bmm86" pod "b867838d-f9a0-45e6-abd3-1d89dabbe6ca" (UID: "b867838d-f9a0-45e6-abd3-1d89dabbe6ca"). InnerVolumeSpecName "kube-api-access-bmm86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:39.604485 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:57:39.604445 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:57:39.623773 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:39.623743 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmm86\" (UniqueName: \"kubernetes.io/projected/b867838d-f9a0-45e6-abd3-1d89dabbe6ca-kube-api-access-bmm86\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:57:40.174318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:40.174282 5047 generic.go:356] "Generic (PLEG): container finished" podID="b867838d-f9a0-45e6-abd3-1d89dabbe6ca" containerID="12929bf02542ecce769c607d3254b4040402d54e6832cb6281e1fa73d7f090c3" exitCode=0 Oct 30 19:57:40.174830 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:40.174385 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" Oct 30 19:57:40.174830 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:40.174425 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" event={"ID":"b867838d-f9a0-45e6-abd3-1d89dabbe6ca","Type":"ContainerDied","Data":"12929bf02542ecce769c607d3254b4040402d54e6832cb6281e1fa73d7f090c3"} Oct 30 19:57:40.174830 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:40.174494 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc" event={"ID":"b867838d-f9a0-45e6-abd3-1d89dabbe6ca","Type":"ContainerDied","Data":"04eb4fdce1de9053cb93a62d3d00c659330ff557646f2447d5b8ba6190e09266"} Oct 30 19:57:40.174830 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:40.174527 5047 scope.go:122] "RemoveContainer" containerID="12929bf02542ecce769c607d3254b4040402d54e6832cb6281e1fa73d7f090c3" Oct 30 19:57:40.186554 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:40.186534 5047 scope.go:122] "RemoveContainer" containerID="12929bf02542ecce769c607d3254b4040402d54e6832cb6281e1fa73d7f090c3" Oct 30 19:57:40.186867 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:57:40.186838 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12929bf02542ecce769c607d3254b4040402d54e6832cb6281e1fa73d7f090c3\": container with ID starting with 12929bf02542ecce769c607d3254b4040402d54e6832cb6281e1fa73d7f090c3 not found: ID does not exist" containerID="12929bf02542ecce769c607d3254b4040402d54e6832cb6281e1fa73d7f090c3" Oct 30 19:57:40.186971 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:40.186878 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12929bf02542ecce769c607d3254b4040402d54e6832cb6281e1fa73d7f090c3"} err="failed to get container status \"12929bf02542ecce769c607d3254b4040402d54e6832cb6281e1fa73d7f090c3\": rpc error: code = NotFound desc = could not find container \"12929bf02542ecce769c607d3254b4040402d54e6832cb6281e1fa73d7f090c3\": container with ID starting with 12929bf02542ecce769c607d3254b4040402d54e6832cb6281e1fa73d7f090c3 not found: ID does not exist" Oct 30 19:57:40.222329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:40.222290 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc"] Oct 30 19:57:40.233997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:40.233959 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-315ea2-54f7fc5cf4-698fc"] Oct 30 19:57:41.051368 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:41.051332 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b867838d-f9a0-45e6-abd3-1d89dabbe6ca" path="/var/lib/kubelet/pods/b867838d-f9a0-45e6-abd3-1d89dabbe6ca/volumes" Oct 30 19:57:46.868318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:46.868273 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f"] Oct 30 19:57:46.876389 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:57:46.876356 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd"] Oct 30 19:58:01.893844 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:01.893781 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" containerID="cri-o://32feb7913287cdf8e8820e7989f5e76ba00093c1dfab6c012277959a51c9ddde" gracePeriod=45 Oct 30 19:58:01.900712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:01.900542 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" containerID="cri-o://82f5bb9a8880d816fd306137e34f425dddcfa24ed4057124ba77762ed566c504" gracePeriod=45 Oct 30 19:58:02.240103 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.240070 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:58:02.245381 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.245357 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:58:02.342449 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.342409 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-model-cache\") pod \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " Oct 30 19:58:02.342648 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.342462 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-home\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-home\") pod \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " Oct 30 19:58:02.342648 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.342501 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-tls-certs\") pod \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " Oct 30 19:58:02.342648 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.342623 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-tmp-dir\") pod \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " Oct 30 19:58:02.342825 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.342701 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-home\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-home\") pod \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " Oct 30 19:58:02.342825 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.342793 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-dshm\") pod \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " Oct 30 19:58:02.342935 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.342833 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-model-cache\") pod \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " Oct 30 19:58:02.342935 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.342865 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-tls-certs\" (UniqueName: \"kubernetes.io/secret/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-tls-certs\") pod \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " Oct 30 19:58:02.343484 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.343451 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-dshm\") pod \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " Oct 30 19:58:02.343484 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.343448 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-tmp-dir" pod "48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" (UID: "48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:02.343678 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.343521 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-kube-api-access-8mgbn\" (UniqueName: \"kubernetes.io/projected/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-kube-api-access-8mgbn\") pod \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\" (UID: \"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f\") " Oct 30 19:58:02.343678 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.343562 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-tmp-dir\") pod \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " Oct 30 19:58:02.343805 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.343711 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-kube-api-access-zfsfj\" (UniqueName: \"kubernetes.io/projected/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-kube-api-access-zfsfj\") pod \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\" (UID: \"1c703bcf-61b7-443b-8b9b-d09d6f11cebc\") " Oct 30 19:58:02.343805 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.343762 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-model-cache" pod "1c703bcf-61b7-443b-8b9b-d09d6f11cebc" (UID: "1c703bcf-61b7-443b-8b9b-d09d6f11cebc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:02.344067 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.344036 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-home" pod "48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" (UID: "48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:02.344643 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.344622 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.344804 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.344789 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.344938 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.344924 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.350033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.349984 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-home" pod "1c703bcf-61b7-443b-8b9b-d09d6f11cebc" (UID: "1c703bcf-61b7-443b-8b9b-d09d6f11cebc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:02.350311 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.350282 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-model-cache" pod "48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" (UID: "48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:02.350551 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.350507 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-tmp-dir" pod "1c703bcf-61b7-443b-8b9b-d09d6f11cebc" (UID: "1c703bcf-61b7-443b-8b9b-d09d6f11cebc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:02.350774 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.350750 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-kube-api-access-zfsfj" pod "1c703bcf-61b7-443b-8b9b-d09d6f11cebc" (UID: "1c703bcf-61b7-443b-8b9b-d09d6f11cebc"). InnerVolumeSpecName "kube-api-access-zfsfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:02.352803 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.352777 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-tls-certs" pod "48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" (UID: "48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:02.352911 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.352849 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-tls-certs" pod "1c703bcf-61b7-443b-8b9b-d09d6f11cebc" (UID: "1c703bcf-61b7-443b-8b9b-d09d6f11cebc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:02.353111 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.353072 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-kube-api-access-8mgbn" pod "48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" (UID: "48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f"). InnerVolumeSpecName "kube-api-access-8mgbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:02.353546 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.353516 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-dshm" pod "48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" (UID: "48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:02.354856 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.354833 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-dshm" pod "1c703bcf-61b7-443b-8b9b-d09d6f11cebc" (UID: "1c703bcf-61b7-443b-8b9b-d09d6f11cebc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:02.446427 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.446217 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.446427 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.446430 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mgbn\" (UniqueName: \"kubernetes.io/projected/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-kube-api-access-8mgbn\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.446708 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.446444 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.446708 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.446459 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfsfj\" (UniqueName: \"kubernetes.io/projected/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-kube-api-access-zfsfj\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.446708 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.446470 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.446708 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.446478 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.446708 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.446485 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c703bcf-61b7-443b-8b9b-d09d6f11cebc-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.446708 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.446494 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.446708 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.446502 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:02.516163 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.516074 5047 generic.go:356] "Generic (PLEG): container finished" podID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerID="82f5bb9a8880d816fd306137e34f425dddcfa24ed4057124ba77762ed566c504" exitCode=0 Oct 30 19:58:02.516163 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.516128 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" Oct 30 19:58:02.516437 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.516162 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" event={"ID":"1c703bcf-61b7-443b-8b9b-d09d6f11cebc","Type":"ContainerDied","Data":"82f5bb9a8880d816fd306137e34f425dddcfa24ed4057124ba77762ed566c504"} Oct 30 19:58:02.516437 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.516216 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" event={"ID":"1c703bcf-61b7-443b-8b9b-d09d6f11cebc","Type":"ContainerDied","Data":"fbfd9ead1674c68363fadd35745b7cf3e6dee27c3fe64fce86324120d8584151"} Oct 30 19:58:02.516437 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.516238 5047 scope.go:122] "RemoveContainer" containerID="82f5bb9a8880d816fd306137e34f425dddcfa24ed4057124ba77762ed566c504" Oct 30 19:58:02.517910 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.517883 5047 generic.go:356] "Generic (PLEG): container finished" podID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerID="32feb7913287cdf8e8820e7989f5e76ba00093c1dfab6c012277959a51c9ddde" exitCode=0 Oct 30 19:58:02.518043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.517923 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" Oct 30 19:58:02.518043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.517967 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" event={"ID":"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f","Type":"ContainerDied","Data":"32feb7913287cdf8e8820e7989f5e76ba00093c1dfab6c012277959a51c9ddde"} Oct 30 19:58:02.518172 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.518046 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f" event={"ID":"48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f","Type":"ContainerDied","Data":"9c117754aff20075158a73384a033cce5c5521468c371503a144653eb08f076f"} Oct 30 19:58:02.529739 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.529648 5047 scope.go:122] "RemoveContainer" containerID="82f5bb9a8880d816fd306137e34f425dddcfa24ed4057124ba77762ed566c504" Oct 30 19:58:02.530110 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:58:02.530086 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f5bb9a8880d816fd306137e34f425dddcfa24ed4057124ba77762ed566c504\": container with ID starting with 82f5bb9a8880d816fd306137e34f425dddcfa24ed4057124ba77762ed566c504 not found: ID does not exist" containerID="82f5bb9a8880d816fd306137e34f425dddcfa24ed4057124ba77762ed566c504" Oct 30 19:58:02.530240 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.530117 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f5bb9a8880d816fd306137e34f425dddcfa24ed4057124ba77762ed566c504"} err="failed to get container status \"82f5bb9a8880d816fd306137e34f425dddcfa24ed4057124ba77762ed566c504\": rpc error: code = NotFound desc = could not find container \"82f5bb9a8880d816fd306137e34f425dddcfa24ed4057124ba77762ed566c504\": container with ID starting with 82f5bb9a8880d816fd306137e34f425dddcfa24ed4057124ba77762ed566c504 not found: ID does not exist" Oct 30 19:58:02.530240 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.530143 5047 scope.go:122] "RemoveContainer" containerID="32feb7913287cdf8e8820e7989f5e76ba00093c1dfab6c012277959a51c9ddde" Oct 30 19:58:02.540300 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.540277 5047 scope.go:122] "RemoveContainer" containerID="32feb7913287cdf8e8820e7989f5e76ba00093c1dfab6c012277959a51c9ddde" Oct 30 19:58:02.540593 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:58:02.540557 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32feb7913287cdf8e8820e7989f5e76ba00093c1dfab6c012277959a51c9ddde\": container with ID starting with 32feb7913287cdf8e8820e7989f5e76ba00093c1dfab6c012277959a51c9ddde not found: ID does not exist" containerID="32feb7913287cdf8e8820e7989f5e76ba00093c1dfab6c012277959a51c9ddde" Oct 30 19:58:02.540693 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.540598 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32feb7913287cdf8e8820e7989f5e76ba00093c1dfab6c012277959a51c9ddde"} err="failed to get container status \"32feb7913287cdf8e8820e7989f5e76ba00093c1dfab6c012277959a51c9ddde\": rpc error: code = NotFound desc = could not find container \"32feb7913287cdf8e8820e7989f5e76ba00093c1dfab6c012277959a51c9ddde\": container with ID starting with 32feb7913287cdf8e8820e7989f5e76ba00093c1dfab6c012277959a51c9ddde not found: ID does not exist" Oct 30 19:58:02.562599 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.562564 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd"] Oct 30 19:58:02.566344 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.566323 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd"] Oct 30 19:58:02.607097 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.607066 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f"] Oct 30 19:58:02.610765 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:02.610738 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-6e0c55/distinct-2-w6-6e0c55-kserve-7f8d856b-w979f"] Oct 30 19:58:03.052265 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:03.052226 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" path="/var/lib/kubelet/pods/1c703bcf-61b7-443b-8b9b-d09d6f11cebc/volumes" Oct 30 19:58:03.052791 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:03.052771 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" path="/var/lib/kubelet/pods/48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f/volumes" Oct 30 19:58:03.220931 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:03.220841 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-6e0c55/distinct-w6-6e0c55-kserve-f956fc7f7-bj8pd" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:09.602816 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:58:09.602781 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:58:12.785876 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:12.785845 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59"] Oct 30 19:58:12.786406 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:12.786079 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" podUID="285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57" containerName="maas-api" containerID="cri-o://7b5d37650c29e1b383e1e05eedc0bb435e9e8672512ee7bf60cba09f7ec92e2e" gracePeriod=30 Oct 30 19:58:13.314811 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.314469 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:58:13.434034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.431588 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-openshift-service-ca\") pod \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\" (UID: \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\") " Oct 30 19:58:13.434034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.431737 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-kube-api-access-ksfg5\" (UniqueName: \"kubernetes.io/projected/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-kube-api-access-ksfg5\") pod \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\" (UID: \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\") " Oct 30 19:58:13.434034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.431866 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-metrics-certs\" (UniqueName: \"kubernetes.io/secret/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-metrics-certs\") pod \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\" (UID: \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\") " Oct 30 19:58:13.434034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.431962 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-maas-api-tls\") pod \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\" (UID: \"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57\") " Oct 30 19:58:13.434034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.433427 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-openshift-service-ca" pod "285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57" (UID: "285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:13.456039 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.454938 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-kube-api-access-ksfg5" pod "285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57" (UID: "285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57"). InnerVolumeSpecName "kube-api-access-ksfg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:13.458499 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.458468 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-maas-api-tls" pod "285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57" (UID: "285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:13.460474 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.460445 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-metrics-certs" pod "285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57" (UID: "285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:13.534189 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.534105 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksfg5\" (UniqueName: \"kubernetes.io/projected/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-kube-api-access-ksfg5\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:13.534189 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.534136 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:13.534189 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.534146 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:13.534189 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.534155 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:13.647521 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.647483 5047 generic.go:356] "Generic (PLEG): container finished" podID="285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57" containerID="7b5d37650c29e1b383e1e05eedc0bb435e9e8672512ee7bf60cba09f7ec92e2e" exitCode=0 Oct 30 19:58:13.647726 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.647544 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" Oct 30 19:58:13.647726 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.647573 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" event={"ID":"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57","Type":"ContainerDied","Data":"7b5d37650c29e1b383e1e05eedc0bb435e9e8672512ee7bf60cba09f7ec92e2e"} Oct 30 19:58:13.647726 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.647648 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59" event={"ID":"285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57","Type":"ContainerDied","Data":"89493e4f72a35dd9dcd82c381a62933e347812f149eb6a33b74a9d8d7f5cfb92"} Oct 30 19:58:13.647726 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.647678 5047 scope.go:122] "RemoveContainer" containerID="7b5d37650c29e1b383e1e05eedc0bb435e9e8672512ee7bf60cba09f7ec92e2e" Oct 30 19:58:13.660719 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.660695 5047 scope.go:122] "RemoveContainer" containerID="7b5d37650c29e1b383e1e05eedc0bb435e9e8672512ee7bf60cba09f7ec92e2e" Oct 30 19:58:13.661042 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:58:13.661019 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5d37650c29e1b383e1e05eedc0bb435e9e8672512ee7bf60cba09f7ec92e2e\": container with ID starting with 7b5d37650c29e1b383e1e05eedc0bb435e9e8672512ee7bf60cba09f7ec92e2e not found: ID does not exist" containerID="7b5d37650c29e1b383e1e05eedc0bb435e9e8672512ee7bf60cba09f7ec92e2e" Oct 30 19:58:13.661111 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.661051 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5d37650c29e1b383e1e05eedc0bb435e9e8672512ee7bf60cba09f7ec92e2e"} err="failed to get container status \"7b5d37650c29e1b383e1e05eedc0bb435e9e8672512ee7bf60cba09f7ec92e2e\": rpc error: code = NotFound desc = could not find container \"7b5d37650c29e1b383e1e05eedc0bb435e9e8672512ee7bf60cba09f7ec92e2e\": container with ID starting with 7b5d37650c29e1b383e1e05eedc0bb435e9e8672512ee7bf60cba09f7ec92e2e not found: ID does not exist" Oct 30 19:58:13.693444 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.693420 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59"] Oct 30 19:58:13.704034 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:13.703996 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-249dd4-7d74878c76-w7p59"] Oct 30 19:58:15.051573 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:15.051538 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57" path="/var/lib/kubelet/pods/285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57/volumes" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.167753 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q"] Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.169810 5047 cpu_manager.go:414] "containerMap: removing container" podUID="87185d96-e36e-47f7-8ea0-dd2e411b01ac" containerName="payload-pre-processing" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.169836 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87185d96-e36e-47f7-8ea0-dd2e411b01ac" containerName="payload-pre-processing" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.169857 5047 cpu_manager.go:414] "containerMap: removing container" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.169868 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.169910 5047 cpu_manager.go:414] "containerMap: removing container" podUID="285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57" containerName="maas-api" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.169918 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57" containerName="maas-api" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.169932 5047 cpu_manager.go:414] "containerMap: removing container" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.169939 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.169986 5047 cpu_manager.go:414] "containerMap: removing container" podUID="c8d7d671-e339-41da-94ff-0c4b553a4147" containerName="payload-processing" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.169993 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d7d671-e339-41da-94ff-0c4b553a4147" containerName="payload-processing" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.170037 5047 cpu_manager.go:414] "containerMap: removing container" podUID="b867838d-f9a0-45e6-abd3-1d89dabbe6ca" containerName="payload-processing" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.170045 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b867838d-f9a0-45e6-abd3-1d89dabbe6ca" containerName="payload-processing" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.170272 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="87185d96-e36e-47f7-8ea0-dd2e411b01ac" containerName="payload-pre-processing" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.170288 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c703bcf-61b7-443b-8b9b-d09d6f11cebc" containerName="main" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.170306 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="285e5b6a-ed6d-437d-ad30-7f6d2d7b9d57" containerName="maas-api" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.170315 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="48dbae1d-a7ed-4bc9-9e21-5dbc87e1851f" containerName="main" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.170332 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d7d671-e339-41da-94ff-0c4b553a4147" containerName="payload-processing" Oct 30 19:58:18.171030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.170349 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="b867838d-f9a0-45e6-abd3-1d89dabbe6ca" containerName="payload-processing" Oct 30 19:58:18.200030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.199905 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q"] Oct 30 19:58:18.200030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.200036 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" Oct 30 19:58:18.226028 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.214339 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-315ea2-dockercfg-srgnr\"" Oct 30 19:58:18.247077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.232703 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-315ea2\"" Oct 30 19:58:18.272356 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.272282 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d764690a-8072-41fb-b021-1c6ad226146f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q\" (UID: \"d764690a-8072-41fb-b021-1c6ad226146f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" Oct 30 19:58:18.272821 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.272791 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mndb\" (UniqueName: \"kubernetes.io/projected/d764690a-8072-41fb-b021-1c6ad226146f-kube-api-access-9mndb\") pod \"payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q\" (UID: \"d764690a-8072-41fb-b021-1c6ad226146f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" Oct 30 19:58:18.316943 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.316890 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l"] Oct 30 19:58:18.324682 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.324653 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" Oct 30 19:58:18.334322 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.334151 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l"] Oct 30 19:58:18.374894 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.374857 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw6n4\" (UniqueName: \"kubernetes.io/projected/3e35030c-45c1-4c40-98e8-8a4a1fe5b72f-kube-api-access-jw6n4\") pod \"payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l\" (UID: \"3e35030c-45c1-4c40-98e8-8a4a1fe5b72f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" Oct 30 19:58:18.375085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.374906 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e35030c-45c1-4c40-98e8-8a4a1fe5b72f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l\" (UID: \"3e35030c-45c1-4c40-98e8-8a4a1fe5b72f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" Oct 30 19:58:18.375085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.375038 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d764690a-8072-41fb-b021-1c6ad226146f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q\" (UID: \"d764690a-8072-41fb-b021-1c6ad226146f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" Oct 30 19:58:18.375211 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.375162 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mndb\" (UniqueName: \"kubernetes.io/projected/d764690a-8072-41fb-b021-1c6ad226146f-kube-api-access-9mndb\") pod \"payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q\" (UID: \"d764690a-8072-41fb-b021-1c6ad226146f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" Oct 30 19:58:18.375840 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.375814 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d764690a-8072-41fb-b021-1c6ad226146f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q\" (UID: \"d764690a-8072-41fb-b021-1c6ad226146f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" Oct 30 19:58:18.394213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.394179 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mndb\" (UniqueName: \"kubernetes.io/projected/d764690a-8072-41fb-b021-1c6ad226146f-kube-api-access-9mndb\") pod \"payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q\" (UID: \"d764690a-8072-41fb-b021-1c6ad226146f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" Oct 30 19:58:18.477053 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.476949 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw6n4\" (UniqueName: \"kubernetes.io/projected/3e35030c-45c1-4c40-98e8-8a4a1fe5b72f-kube-api-access-jw6n4\") pod \"payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l\" (UID: \"3e35030c-45c1-4c40-98e8-8a4a1fe5b72f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" Oct 30 19:58:18.477053 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.476990 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e35030c-45c1-4c40-98e8-8a4a1fe5b72f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l\" (UID: \"3e35030c-45c1-4c40-98e8-8a4a1fe5b72f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" Oct 30 19:58:18.477819 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.477796 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e35030c-45c1-4c40-98e8-8a4a1fe5b72f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l\" (UID: \"3e35030c-45c1-4c40-98e8-8a4a1fe5b72f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" Oct 30 19:58:18.494201 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.494166 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw6n4\" (UniqueName: \"kubernetes.io/projected/3e35030c-45c1-4c40-98e8-8a4a1fe5b72f-kube-api-access-jw6n4\") pod \"payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l\" (UID: \"3e35030c-45c1-4c40-98e8-8a4a1fe5b72f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" Oct 30 19:58:18.557277 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.557241 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" Oct 30 19:58:18.649465 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.649432 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" Oct 30 19:58:18.863961 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.863929 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q"] Oct 30 19:58:18.864294 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:58:18.864267 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd764690a_8072_41fb_b021_1c6ad226146f.slice/crio-66d54f9a790ee6556f8a0cad15cd802dd967e698472da69459721322c9fc6645 WatchSource:0}: Error finding container 66d54f9a790ee6556f8a0cad15cd802dd967e698472da69459721322c9fc6645: Status 404 returned error can't find the container with id 66d54f9a790ee6556f8a0cad15cd802dd967e698472da69459721322c9fc6645 Oct 30 19:58:18.969191 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:18.969160 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l"] Oct 30 19:58:18.969689 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:58:18.969655 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e35030c_45c1_4c40_98e8_8a4a1fe5b72f.slice/crio-020af34bcb336ff1d4d6e36c592992d76bfb975def50527f90a8900ead6b7683 WatchSource:0}: Error finding container 020af34bcb336ff1d4d6e36c592992d76bfb975def50527f90a8900ead6b7683: Status 404 returned error can't find the container with id 020af34bcb336ff1d4d6e36c592992d76bfb975def50527f90a8900ead6b7683 Oct 30 19:58:19.719797 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:19.719745 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" event={"ID":"3e35030c-45c1-4c40-98e8-8a4a1fe5b72f","Type":"ContainerStarted","Data":"313adebc5edec1e24855657b353a42edf64a74bb5cdd7ac5e8d052c395262c18"} Oct 30 19:58:19.720294 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:19.719836 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" Oct 30 19:58:19.720294 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:19.719863 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" event={"ID":"3e35030c-45c1-4c40-98e8-8a4a1fe5b72f","Type":"ContainerStarted","Data":"020af34bcb336ff1d4d6e36c592992d76bfb975def50527f90a8900ead6b7683"} Oct 30 19:58:19.722273 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:19.722242 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" event={"ID":"d764690a-8072-41fb-b021-1c6ad226146f","Type":"ContainerStarted","Data":"9ddd2a1cdcdf9190bea59a337f5a98b958b185e47c4c48d8c6c1a7d58e9aeac9"} Oct 30 19:58:19.722432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:19.722301 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" event={"ID":"d764690a-8072-41fb-b021-1c6ad226146f","Type":"ContainerStarted","Data":"66d54f9a790ee6556f8a0cad15cd802dd967e698472da69459721322c9fc6645"} Oct 30 19:58:19.722432 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:19.722329 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" Oct 30 19:58:19.756369 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:19.756311 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" podStartSLOduration=1.756296139 podStartE2EDuration="1.756296139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:19.754039841 +0000 UTC m=+3252.828495199" watchObservedRunningTime="2026-10-30 19:58:19.756296139 +0000 UTC m=+3252.830751536" Oct 30 19:58:19.790351 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:19.790285 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" podStartSLOduration=1.790265481 podStartE2EDuration="1.790265481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:19.788422806 +0000 UTC m=+3252.862878165" watchObservedRunningTime="2026-10-30 19:58:19.790265481 +0000 UTC m=+3252.864720841" Oct 30 19:58:21.023892 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:21.023838 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt"] Oct 30 19:58:21.024410 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:21.024188 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" podUID="8fd1d5de-7f02-4d79-a762-ee3d77c86bba" containerName="istio-proxy" containerID="cri-o://4efff2cf40c7ba3c3e1463b96984fbe11730c195277fec9aa0b96347107d893b" gracePeriod=30 Oct 30 19:58:26.702342 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.702316 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:58:26.803489 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.803397 5047 generic.go:356] "Generic (PLEG): container finished" podID="8fd1d5de-7f02-4d79-a762-ee3d77c86bba" containerID="4efff2cf40c7ba3c3e1463b96984fbe11730c195277fec9aa0b96347107d893b" exitCode=0 Oct 30 19:58:26.803489 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.803452 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" Oct 30 19:58:26.803735 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.803481 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" event={"ID":"8fd1d5de-7f02-4d79-a762-ee3d77c86bba","Type":"ContainerDied","Data":"4efff2cf40c7ba3c3e1463b96984fbe11730c195277fec9aa0b96347107d893b"} Oct 30 19:58:26.803735 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.803539 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt" event={"ID":"8fd1d5de-7f02-4d79-a762-ee3d77c86bba","Type":"ContainerDied","Data":"9a7154ec79213135dd3570487ec91a2cda3b2f5980de9748183954f23a11fa64"} Oct 30 19:58:26.803735 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.803562 5047 scope.go:122] "RemoveContainer" containerID="4efff2cf40c7ba3c3e1463b96984fbe11730c195277fec9aa0b96347107d893b" Oct 30 19:58:26.817226 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.817124 5047 scope.go:122] "RemoveContainer" containerID="56763c4475020b081e4a70044ea983e9e32616c596850747719576485bb8d11c" Oct 30 19:58:26.825774 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.825733 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-kube-api-access-54lrn\" (UniqueName: \"kubernetes.io/projected/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-kube-api-access-54lrn\") pod \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " Oct 30 19:58:26.825922 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.825816 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-data\") pod \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " Oct 30 19:58:26.825922 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.825910 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-workload-socket\") pod \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " Oct 30 19:58:26.826071 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.826036 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-token\" (UniqueName: \"kubernetes.io/projected/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-token\") pod \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " Oct 30 19:58:26.826147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.826084 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-workload-certs\") pod \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " Oct 30 19:58:26.826147 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.826127 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-podinfo\") pod \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " Oct 30 19:58:26.826261 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.826199 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-credential-socket\") pod \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " Oct 30 19:58:26.826261 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.826246 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-envoy\") pod \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " Oct 30 19:58:26.826370 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.826281 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istiod-ca-cert\") pod \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\" (UID: \"8fd1d5de-7f02-4d79-a762-ee3d77c86bba\") " Oct 30 19:58:26.826438 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.826368 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-workload-socket" pod "8fd1d5de-7f02-4d79-a762-ee3d77c86bba" (UID: "8fd1d5de-7f02-4d79-a762-ee3d77c86bba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:26.827036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.826804 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-credential-socket" pod "8fd1d5de-7f02-4d79-a762-ee3d77c86bba" (UID: "8fd1d5de-7f02-4d79-a762-ee3d77c86bba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:26.827036 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.826991 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-workload-certs" pod "8fd1d5de-7f02-4d79-a762-ee3d77c86bba" (UID: "8fd1d5de-7f02-4d79-a762-ee3d77c86bba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:26.828033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.827436 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:26.828033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.827466 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:26.828033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.827481 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:26.828033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.827796 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istiod-ca-cert" pod "8fd1d5de-7f02-4d79-a762-ee3d77c86bba" (UID: "8fd1d5de-7f02-4d79-a762-ee3d77c86bba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:26.828033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.827974 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-data" pod "8fd1d5de-7f02-4d79-a762-ee3d77c86bba" (UID: "8fd1d5de-7f02-4d79-a762-ee3d77c86bba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:26.828807 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.828774 5047 scope.go:122] "RemoveContainer" containerID="4efff2cf40c7ba3c3e1463b96984fbe11730c195277fec9aa0b96347107d893b" Oct 30 19:58:26.829241 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:58:26.829209 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4efff2cf40c7ba3c3e1463b96984fbe11730c195277fec9aa0b96347107d893b\": container with ID starting with 4efff2cf40c7ba3c3e1463b96984fbe11730c195277fec9aa0b96347107d893b not found: ID does not exist" containerID="4efff2cf40c7ba3c3e1463b96984fbe11730c195277fec9aa0b96347107d893b" Oct 30 19:58:26.829359 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.829253 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efff2cf40c7ba3c3e1463b96984fbe11730c195277fec9aa0b96347107d893b"} err="failed to get container status \"4efff2cf40c7ba3c3e1463b96984fbe11730c195277fec9aa0b96347107d893b\": rpc error: code = NotFound desc = could not find container \"4efff2cf40c7ba3c3e1463b96984fbe11730c195277fec9aa0b96347107d893b\": container with ID starting with 4efff2cf40c7ba3c3e1463b96984fbe11730c195277fec9aa0b96347107d893b not found: ID does not exist" Oct 30 19:58:26.829359 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.829282 5047 scope.go:122] "RemoveContainer" containerID="56763c4475020b081e4a70044ea983e9e32616c596850747719576485bb8d11c" Oct 30 19:58:26.829637 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:58:26.829612 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56763c4475020b081e4a70044ea983e9e32616c596850747719576485bb8d11c\": container with ID starting with 56763c4475020b081e4a70044ea983e9e32616c596850747719576485bb8d11c not found: ID does not exist" containerID="56763c4475020b081e4a70044ea983e9e32616c596850747719576485bb8d11c" Oct 30 19:58:26.829738 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.829645 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56763c4475020b081e4a70044ea983e9e32616c596850747719576485bb8d11c"} err="failed to get container status \"56763c4475020b081e4a70044ea983e9e32616c596850747719576485bb8d11c\": rpc error: code = NotFound desc = could not find container \"56763c4475020b081e4a70044ea983e9e32616c596850747719576485bb8d11c\": container with ID starting with 56763c4475020b081e4a70044ea983e9e32616c596850747719576485bb8d11c not found: ID does not exist" Oct 30 19:58:26.830666 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.830627 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-envoy" pod "8fd1d5de-7f02-4d79-a762-ee3d77c86bba" (UID: "8fd1d5de-7f02-4d79-a762-ee3d77c86bba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:26.832288 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.832264 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-kube-api-access-54lrn" pod "8fd1d5de-7f02-4d79-a762-ee3d77c86bba" (UID: "8fd1d5de-7f02-4d79-a762-ee3d77c86bba"). InnerVolumeSpecName "kube-api-access-54lrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:26.832510 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.832477 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-podinfo" pod "8fd1d5de-7f02-4d79-a762-ee3d77c86bba" (UID: "8fd1d5de-7f02-4d79-a762-ee3d77c86bba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:26.832600 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.832580 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-token" pod "8fd1d5de-7f02-4d79-a762-ee3d77c86bba" (UID: "8fd1d5de-7f02-4d79-a762-ee3d77c86bba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:26.928913 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.928877 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54lrn\" (UniqueName: \"kubernetes.io/projected/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-kube-api-access-54lrn\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:26.928913 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.928911 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:26.929144 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.928924 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:26.929144 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.928932 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:26.929144 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.928940 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:26.929144 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:26.928949 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd1d5de-7f02-4d79-a762-ee3d77c86bba-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:27.143203 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:27.143131 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt"] Oct 30 19:58:27.153743 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:27.153701 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-249dd4-openshift-default-59f7849b69-97plt"] Oct 30 19:58:29.071501 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:29.071464 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fd1d5de-7f02-4d79-a762-ee3d77c86bba" path="/var/lib/kubelet/pods/8fd1d5de-7f02-4d79-a762-ee3d77c86bba/volumes" Oct 30 19:58:29.776860 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:29.776819 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v"] Oct 30 19:58:29.777105 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:29.777057 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" podUID="3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" containerName="istio-proxy" containerID="cri-o://b5417d430c6c552de60775d57dd57543b0e54c0ef46308cce92b83360ee62d45" gracePeriod=30 Oct 30 19:58:30.732821 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:30.732788 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" Oct 30 19:58:30.733331 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:30.733214 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" Oct 30 19:58:33.299675 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:33.299640 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx"] Oct 30 19:58:33.300169 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:33.299890 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" podUID="3a7dbb86-134e-482d-b8f1-17a5105fdca6" containerName="maas-api" containerID="cri-o://8f53add73de105a864a9bcd6d0280a929c3b8bdcbf820b6b2c19835544ae34a2" gracePeriod=30 Oct 30 19:58:33.479886 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:33.479847 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5"] Oct 30 19:58:33.480161 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:33.480128 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" podUID="69575b4d-726e-49dc-b0d8-50be11ea88db" containerName="payload-processing" containerID="cri-o://2d673fd2d9008f40ffce6c82d342b21a957680c0e235224ae57c199464a0f2cf" gracePeriod=30 Oct 30 19:58:33.753318 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:33.752632 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:58:33.870043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:33.869825 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a7dbb86-134e-482d-b8f1-17a5105fdca6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a7dbb86-134e-482d-b8f1-17a5105fdca6-maas-api-tls\") pod \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\" (UID: \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\") " Oct 30 19:58:33.880043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:33.872117 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a7dbb86-134e-482d-b8f1-17a5105fdca6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a7dbb86-134e-482d-b8f1-17a5105fdca6-metrics-certs\") pod \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\" (UID: \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\") " Oct 30 19:58:33.909051 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:33.907052 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a7dbb86-134e-482d-b8f1-17a5105fdca6-kube-api-access-r76jg\" (UniqueName: \"kubernetes.io/projected/3a7dbb86-134e-482d-b8f1-17a5105fdca6-kube-api-access-r76jg\") pod \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\" (UID: \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\") " Oct 30 19:58:33.909051 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:33.907186 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a7dbb86-134e-482d-b8f1-17a5105fdca6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7dbb86-134e-482d-b8f1-17a5105fdca6-openshift-service-ca\") pod \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\" (UID: \"3a7dbb86-134e-482d-b8f1-17a5105fdca6\") " Oct 30 19:58:33.909051 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:33.907578 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a7dbb86-134e-482d-b8f1-17a5105fdca6-openshift-service-ca" pod "3a7dbb86-134e-482d-b8f1-17a5105fdca6" (UID: "3a7dbb86-134e-482d-b8f1-17a5105fdca6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:33.909051 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:33.908412 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7dbb86-134e-482d-b8f1-17a5105fdca6-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:33.967213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:33.957158 5047 generic.go:356] "Generic (PLEG): container finished" podID="3a7dbb86-134e-482d-b8f1-17a5105fdca6" containerID="8f53add73de105a864a9bcd6d0280a929c3b8bdcbf820b6b2c19835544ae34a2" exitCode=0 Oct 30 19:58:33.967213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:33.957352 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" event={"ID":"3a7dbb86-134e-482d-b8f1-17a5105fdca6","Type":"ContainerDied","Data":"8f53add73de105a864a9bcd6d0280a929c3b8bdcbf820b6b2c19835544ae34a2"} Oct 30 19:58:33.967213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:33.957407 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" event={"ID":"3a7dbb86-134e-482d-b8f1-17a5105fdca6","Type":"ContainerDied","Data":"ab38099f695589e3d5a7ed9bf384307f746b47bb73633740d62974db96bea88b"} Oct 30 19:58:33.967213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:33.957438 5047 scope.go:122] "RemoveContainer" containerID="8f53add73de105a864a9bcd6d0280a929c3b8bdcbf820b6b2c19835544ae34a2" Oct 30 19:58:33.967213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:33.957584 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx" Oct 30 19:58:34.025562 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.008019 5047 generic.go:356] "Generic (PLEG): container finished" podID="69575b4d-726e-49dc-b0d8-50be11ea88db" containerID="2d673fd2d9008f40ffce6c82d342b21a957680c0e235224ae57c199464a0f2cf" exitCode=0 Oct 30 19:58:34.025562 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.008120 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" event={"ID":"69575b4d-726e-49dc-b0d8-50be11ea88db","Type":"ContainerDied","Data":"2d673fd2d9008f40ffce6c82d342b21a957680c0e235224ae57c199464a0f2cf"} Oct 30 19:58:34.029422 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.029362 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7dbb86-134e-482d-b8f1-17a5105fdca6-kube-api-access-r76jg" pod "3a7dbb86-134e-482d-b8f1-17a5105fdca6" (UID: "3a7dbb86-134e-482d-b8f1-17a5105fdca6"). InnerVolumeSpecName "kube-api-access-r76jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:34.037213 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.037184 5047 scope.go:122] "RemoveContainer" containerID="8f53add73de105a864a9bcd6d0280a929c3b8bdcbf820b6b2c19835544ae34a2" Oct 30 19:58:34.037953 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:58:34.037926 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f53add73de105a864a9bcd6d0280a929c3b8bdcbf820b6b2c19835544ae34a2\": container with ID starting with 8f53add73de105a864a9bcd6d0280a929c3b8bdcbf820b6b2c19835544ae34a2 not found: ID does not exist" containerID="8f53add73de105a864a9bcd6d0280a929c3b8bdcbf820b6b2c19835544ae34a2" Oct 30 19:58:34.038251 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.038222 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f53add73de105a864a9bcd6d0280a929c3b8bdcbf820b6b2c19835544ae34a2"} err="failed to get container status \"8f53add73de105a864a9bcd6d0280a929c3b8bdcbf820b6b2c19835544ae34a2\": rpc error: code = NotFound desc = could not find container \"8f53add73de105a864a9bcd6d0280a929c3b8bdcbf820b6b2c19835544ae34a2\": container with ID starting with 8f53add73de105a864a9bcd6d0280a929c3b8bdcbf820b6b2c19835544ae34a2 not found: ID does not exist" Oct 30 19:58:34.044601 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.044558 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a7dbb86-134e-482d-b8f1-17a5105fdca6-metrics-certs" pod "3a7dbb86-134e-482d-b8f1-17a5105fdca6" (UID: "3a7dbb86-134e-482d-b8f1-17a5105fdca6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:34.047072 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.047034 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a7dbb86-134e-482d-b8f1-17a5105fdca6-maas-api-tls" pod "3a7dbb86-134e-482d-b8f1-17a5105fdca6" (UID: "3a7dbb86-134e-482d-b8f1-17a5105fdca6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:34.068044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.066412 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" Oct 30 19:58:34.129180 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.129079 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69575b4d-726e-49dc-b0d8-50be11ea88db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69575b4d-726e-49dc-b0d8-50be11ea88db-plugins-config-volume\") pod \"69575b4d-726e-49dc-b0d8-50be11ea88db\" (UID: \"69575b4d-726e-49dc-b0d8-50be11ea88db\") " Oct 30 19:58:34.129362 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.129208 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69575b4d-726e-49dc-b0d8-50be11ea88db-kube-api-access-jjdd7\" (UniqueName: \"kubernetes.io/projected/69575b4d-726e-49dc-b0d8-50be11ea88db-kube-api-access-jjdd7\") pod \"69575b4d-726e-49dc-b0d8-50be11ea88db\" (UID: \"69575b4d-726e-49dc-b0d8-50be11ea88db\") " Oct 30 19:58:34.129466 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.129445 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69575b4d-726e-49dc-b0d8-50be11ea88db-plugins-config-volume" pod "69575b4d-726e-49dc-b0d8-50be11ea88db" (UID: "69575b4d-726e-49dc-b0d8-50be11ea88db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:34.130233 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.130142 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69575b4d-726e-49dc-b0d8-50be11ea88db-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.130233 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.130167 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r76jg\" (UniqueName: \"kubernetes.io/projected/3a7dbb86-134e-482d-b8f1-17a5105fdca6-kube-api-access-r76jg\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.130233 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.130180 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a7dbb86-134e-482d-b8f1-17a5105fdca6-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.130233 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.130193 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a7dbb86-134e-482d-b8f1-17a5105fdca6-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.131788 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.131761 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69575b4d-726e-49dc-b0d8-50be11ea88db-kube-api-access-jjdd7" pod "69575b4d-726e-49dc-b0d8-50be11ea88db" (UID: "69575b4d-726e-49dc-b0d8-50be11ea88db"). InnerVolumeSpecName "kube-api-access-jjdd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:34.236367 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.236332 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjdd7\" (UniqueName: \"kubernetes.io/projected/69575b4d-726e-49dc-b0d8-50be11ea88db-kube-api-access-jjdd7\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:34.305982 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.305944 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx"] Oct 30 19:58:34.309731 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:34.309697 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b36ad-f7b77f46b-vtxsx"] Oct 30 19:58:35.019060 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.019024 5047 generic.go:356] "Generic (PLEG): container finished" podID="3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" containerID="b5417d430c6c552de60775d57dd57543b0e54c0ef46308cce92b83360ee62d45" exitCode=0 Oct 30 19:58:35.019268 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.019062 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" event={"ID":"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2","Type":"ContainerDied","Data":"b5417d430c6c552de60775d57dd57543b0e54c0ef46308cce92b83360ee62d45"} Oct 30 19:58:35.020859 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.020772 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" Oct 30 19:58:35.020859 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.020795 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5" event={"ID":"69575b4d-726e-49dc-b0d8-50be11ea88db","Type":"ContainerDied","Data":"8e9300f0ae7bdeba9bc1a4c90614557c3a9432de630525a40b9ddd9a306aa990"} Oct 30 19:58:35.020859 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.020859 5047 scope.go:122] "RemoveContainer" containerID="2d673fd2d9008f40ffce6c82d342b21a957680c0e235224ae57c199464a0f2cf" Oct 30 19:58:35.052043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.051971 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a7dbb86-134e-482d-b8f1-17a5105fdca6" path="/var/lib/kubelet/pods/3a7dbb86-134e-482d-b8f1-17a5105fdca6/volumes" Oct 30 19:58:35.068926 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.068897 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5"] Oct 30 19:58:35.076728 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.076703 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b36ad-7557679f96-wdjr5"] Oct 30 19:58:35.085386 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.085362 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:58:35.172876 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.172837 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-data\") pod \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " Oct 30 19:58:35.173073 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.172924 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-token\" (UniqueName: \"kubernetes.io/projected/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-token\") pod \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " Oct 30 19:58:35.173073 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.172951 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istiod-ca-cert\") pod \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " Oct 30 19:58:35.173216 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.173164 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-credential-socket\") pod \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " Oct 30 19:58:35.173452 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.173422 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-workload-socket\") pod \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " Oct 30 19:58:35.173580 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.173483 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-workload-certs\") pod \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " Oct 30 19:58:35.173580 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.173424 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istiod-ca-cert" pod "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" (UID: "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:35.173580 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.173565 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-envoy\") pod \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " Oct 30 19:58:35.173762 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.173632 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-kube-api-access-vtk2n\" (UniqueName: \"kubernetes.io/projected/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-kube-api-access-vtk2n\") pod \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " Oct 30 19:58:35.173762 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.173672 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-podinfo\") pod \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\" (UID: \"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2\") " Oct 30 19:58:35.173762 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.173685 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-credential-socket" pod "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" (UID: "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:35.173947 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.173789 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-workload-certs" pod "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" (UID: "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:35.174162 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.174134 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-workload-socket" pod "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" (UID: "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:35.174324 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.174271 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-data" pod "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" (UID: "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:35.174600 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.174575 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.174712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.174602 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.174712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.174617 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.174712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.174631 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.174712 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.174645 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.176239 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.176208 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-token" pod "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" (UID: "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:35.177405 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.177378 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-kube-api-access-vtk2n" pod "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" (UID: "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2"). InnerVolumeSpecName "kube-api-access-vtk2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:35.177405 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.177390 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-envoy" pod "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" (UID: "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:35.177528 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.177433 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-podinfo" pod "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" (UID: "3cd19e5f-186b-4b69-b6b5-99e53d3e62a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:35.276777 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.276736 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.276777 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.276772 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtk2n\" (UniqueName: \"kubernetes.io/projected/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-kube-api-access-vtk2n\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.276777 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.276782 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:35.277043 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:35.276791 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:36.032742 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:36.032701 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" event={"ID":"3cd19e5f-186b-4b69-b6b5-99e53d3e62a2","Type":"ContainerDied","Data":"717d4ad6d4e27fd6187a598dd49e82f29d8b39b9d93509cb23b5fcc692add666"} Oct 30 19:58:36.032742 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:36.032746 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v" Oct 30 19:58:36.033603 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:36.032809 5047 scope.go:122] "RemoveContainer" containerID="b5417d430c6c552de60775d57dd57543b0e54c0ef46308cce92b83360ee62d45" Oct 30 19:58:36.101821 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:36.101786 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v"] Oct 30 19:58:36.113851 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:36.113823 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-287ea2-openshift-default-76f4868567-qkr4v"] Oct 30 19:58:37.051886 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:37.051845 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" path="/var/lib/kubelet/pods/3cd19e5f-186b-4b69-b6b5-99e53d3e62a2/volumes" Oct 30 19:58:37.052559 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:37.052536 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69575b4d-726e-49dc-b0d8-50be11ea88db" path="/var/lib/kubelet/pods/69575b4d-726e-49dc-b0d8-50be11ea88db/volumes" Oct 30 19:58:38.050977 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.050891 5047 generic.go:356] "Generic (PLEG): container finished" podID="8856e033-793e-4e57-abbf-ae7ae07cbff4" containerID="b181b06d3a5acaf3e8e45a05e781d6dc9fb86f6b8931d5432014a39e792bd0ff" exitCode=0 Oct 30 19:58:38.050977 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.050964 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" event={"ID":"8856e033-793e-4e57-abbf-ae7ae07cbff4","Type":"ContainerDied","Data":"b181b06d3a5acaf3e8e45a05e781d6dc9fb86f6b8931d5432014a39e792bd0ff"} Oct 30 19:58:38.051443 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.051424 5047 scope.go:122] "RemoveContainer" containerID="b181b06d3a5acaf3e8e45a05e781d6dc9fb86f6b8931d5432014a39e792bd0ff" Oct 30 19:58:38.053205 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.053177 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:58:38.053485 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.053244 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:58:38.053485 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.053276 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:58:38.165444 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.165410 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv"] Oct 30 19:58:38.166532 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.166506 5047 cpu_manager.go:414] "containerMap: removing container" podUID="3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" containerName="istio-proxy" Oct 30 19:58:38.166532 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.166531 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" containerName="istio-proxy" Oct 30 19:58:38.166754 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.166564 5047 cpu_manager.go:414] "containerMap: removing container" podUID="8fd1d5de-7f02-4d79-a762-ee3d77c86bba" containerName="istio-proxy" Oct 30 19:58:38.166754 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.166571 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fd1d5de-7f02-4d79-a762-ee3d77c86bba" containerName="istio-proxy" Oct 30 19:58:38.166754 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.166589 5047 cpu_manager.go:414] "containerMap: removing container" podUID="69575b4d-726e-49dc-b0d8-50be11ea88db" containerName="payload-processing" Oct 30 19:58:38.166754 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.166597 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69575b4d-726e-49dc-b0d8-50be11ea88db" containerName="payload-processing" Oct 30 19:58:38.166754 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.166616 5047 cpu_manager.go:414] "containerMap: removing container" podUID="3a7dbb86-134e-482d-b8f1-17a5105fdca6" containerName="maas-api" Oct 30 19:58:38.166754 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.166624 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a7dbb86-134e-482d-b8f1-17a5105fdca6" containerName="maas-api" Oct 30 19:58:38.166754 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.166643 5047 cpu_manager.go:414] "containerMap: removing container" podUID="8fd1d5de-7f02-4d79-a762-ee3d77c86bba" containerName="istio-proxy" Oct 30 19:58:38.166754 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.166650 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fd1d5de-7f02-4d79-a762-ee3d77c86bba" containerName="istio-proxy" Oct 30 19:58:38.167220 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.166900 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd19e5f-186b-4b69-b6b5-99e53d3e62a2" containerName="istio-proxy" Oct 30 19:58:38.167220 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.166924 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fd1d5de-7f02-4d79-a762-ee3d77c86bba" containerName="istio-proxy" Oct 30 19:58:38.167220 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.166934 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="69575b4d-726e-49dc-b0d8-50be11ea88db" containerName="payload-processing" Oct 30 19:58:38.167220 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.166946 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a7dbb86-134e-482d-b8f1-17a5105fdca6" containerName="maas-api" Oct 30 19:58:38.170851 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.170831 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" Oct 30 19:58:38.191626 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.186982 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e430a0\"" Oct 30 19:58:38.193197 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.193171 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv"] Oct 30 19:58:38.227633 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.227586 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9566ea28-5afe-4ae9-87d9-c1444aa8aeb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv\" (UID: \"9566ea28-5afe-4ae9-87d9-c1444aa8aeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" Oct 30 19:58:38.227834 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.227739 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c887m\" (UniqueName: \"kubernetes.io/projected/9566ea28-5afe-4ae9-87d9-c1444aa8aeb3-kube-api-access-c887m\") pod \"payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv\" (UID: \"9566ea28-5afe-4ae9-87d9-c1444aa8aeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" Oct 30 19:58:38.264565 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.263111 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e430a0-dockercfg-kgt4g\"" Oct 30 19:58:38.270600 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.270552 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l"] Oct 30 19:58:38.272688 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.272653 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fd1d5de-7f02-4d79-a762-ee3d77c86bba" containerName="istio-proxy" Oct 30 19:58:38.281095 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.280953 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" Oct 30 19:58:38.358090 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.355484 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l"] Oct 30 19:58:38.366085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.362521 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzdt4\" (UniqueName: \"kubernetes.io/projected/48a34ff4-bd18-4a74-a78f-368c1ac5c651-kube-api-access-gzdt4\") pod \"payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l\" (UID: \"48a34ff4-bd18-4a74-a78f-368c1ac5c651\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" Oct 30 19:58:38.366085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.362614 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9566ea28-5afe-4ae9-87d9-c1444aa8aeb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv\" (UID: \"9566ea28-5afe-4ae9-87d9-c1444aa8aeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" Oct 30 19:58:38.366085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.362791 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c887m\" (UniqueName: \"kubernetes.io/projected/9566ea28-5afe-4ae9-87d9-c1444aa8aeb3-kube-api-access-c887m\") pod \"payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv\" (UID: \"9566ea28-5afe-4ae9-87d9-c1444aa8aeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" Oct 30 19:58:38.366085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.362837 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a34ff4-bd18-4a74-a78f-368c1ac5c651-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l\" (UID: \"48a34ff4-bd18-4a74-a78f-368c1ac5c651\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" Oct 30 19:58:38.366085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.364165 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9566ea28-5afe-4ae9-87d9-c1444aa8aeb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv\" (UID: \"9566ea28-5afe-4ae9-87d9-c1444aa8aeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" Oct 30 19:58:38.402850 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.402670 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c887m\" (UniqueName: \"kubernetes.io/projected/9566ea28-5afe-4ae9-87d9-c1444aa8aeb3-kube-api-access-c887m\") pod \"payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv\" (UID: \"9566ea28-5afe-4ae9-87d9-c1444aa8aeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" Oct 30 19:58:38.467052 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.466959 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdt4\" (UniqueName: \"kubernetes.io/projected/48a34ff4-bd18-4a74-a78f-368c1ac5c651-kube-api-access-gzdt4\") pod \"payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l\" (UID: \"48a34ff4-bd18-4a74-a78f-368c1ac5c651\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" Oct 30 19:58:38.467236 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.467152 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a34ff4-bd18-4a74-a78f-368c1ac5c651-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l\" (UID: \"48a34ff4-bd18-4a74-a78f-368c1ac5c651\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" Oct 30 19:58:38.468874 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.468845 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a34ff4-bd18-4a74-a78f-368c1ac5c651-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l\" (UID: \"48a34ff4-bd18-4a74-a78f-368c1ac5c651\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" Oct 30 19:58:38.491252 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.491220 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzdt4\" (UniqueName: \"kubernetes.io/projected/48a34ff4-bd18-4a74-a78f-368c1ac5c651-kube-api-access-gzdt4\") pod \"payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l\" (UID: \"48a34ff4-bd18-4a74-a78f-368c1ac5c651\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" Oct 30 19:58:38.500223 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.500194 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" Oct 30 19:58:38.777893 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.777810 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" Oct 30 19:58:38.818796 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:58:38.818764 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9566ea28_5afe_4ae9_87d9_c1444aa8aeb3.slice/crio-5e2a5becdf3121ca5b615598fe2fac5cb3cfdcd781ce51d1a630db3428fb7486 WatchSource:0}: Error finding container 5e2a5becdf3121ca5b615598fe2fac5cb3cfdcd781ce51d1a630db3428fb7486: Status 404 returned error can't find the container with id 5e2a5becdf3121ca5b615598fe2fac5cb3cfdcd781ce51d1a630db3428fb7486 Oct 30 19:58:38.819384 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:38.819369 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv"] Oct 30 19:58:39.061033 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:39.060940 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" event={"ID":"8856e033-793e-4e57-abbf-ae7ae07cbff4","Type":"ContainerStarted","Data":"5ba2ed8723e740c875fb16073eee4548419ce02f38fba3ea0ea9ed6ae2a83446"} Oct 30 19:58:39.063414 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:39.063386 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" event={"ID":"9566ea28-5afe-4ae9-87d9-c1444aa8aeb3","Type":"ContainerStarted","Data":"18584eedb2d4988aac5058df2aaa5fe162b8922b7a8f479ed07f0214cc533071"} Oct 30 19:58:39.063541 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:39.063429 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" event={"ID":"9566ea28-5afe-4ae9-87d9-c1444aa8aeb3","Type":"ContainerStarted","Data":"5e2a5becdf3121ca5b615598fe2fac5cb3cfdcd781ce51d1a630db3428fb7486"} Oct 30 19:58:39.063541 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:39.063451 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" Oct 30 19:58:39.085227 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:39.085200 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l"] Oct 30 19:58:39.085362 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:58:39.085338 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a34ff4_bd18_4a74_a78f_368c1ac5c651.slice/crio-5414a2b74c0199cf8d0a1f08e5301644c07667921591de8fcbc4b78f1a3ccdd1 WatchSource:0}: Error finding container 5414a2b74c0199cf8d0a1f08e5301644c07667921591de8fcbc4b78f1a3ccdd1: Status 404 returned error can't find the container with id 5414a2b74c0199cf8d0a1f08e5301644c07667921591de8fcbc4b78f1a3ccdd1 Oct 30 19:58:39.134628 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:39.134572 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" podStartSLOduration=1.134551709 podStartE2EDuration="1.134551709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:39.130250278 +0000 UTC m=+3272.204705637" watchObservedRunningTime="2026-10-30 19:58:39.134551709 +0000 UTC m=+3272.209007082" Oct 30 19:58:39.602930 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:58:39.602874 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:58:39.870086 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:39.869453 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:58:39.887030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:39.878144 5047 patch_prober.go:29] interesting pod/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 19:58:39.887030 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:39.878214 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" podUID="8856e033-793e-4e57-abbf-ae7ae07cbff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 19:58:40.080581 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:40.080121 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" event={"ID":"48a34ff4-bd18-4a74-a78f-368c1ac5c651","Type":"ContainerStarted","Data":"3646bfa1c7216c2899f1afa4a0521e7132dee17778d8224e25cbb5ecece7b011"} Oct 30 19:58:40.080581 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:40.080187 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" event={"ID":"48a34ff4-bd18-4a74-a78f-368c1ac5c651","Type":"ContainerStarted","Data":"5414a2b74c0199cf8d0a1f08e5301644c07667921591de8fcbc4b78f1a3ccdd1"} Oct 30 19:58:40.081292 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:40.081265 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" Oct 30 19:58:40.118699 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:40.118659 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" podStartSLOduration=2.118646834 podStartE2EDuration="2.118646834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:40.115244352 +0000 UTC m=+3273.189699710" watchObservedRunningTime="2026-10-30 19:58:40.118646834 +0000 UTC m=+3273.193102193" Oct 30 19:58:40.866560 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:40.866519 5047 patch_prober.go:29] interesting pod/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 19:58:40.866760 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:40.866576 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" podUID="8856e033-793e-4e57-abbf-ae7ae07cbff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 19:58:40.993519 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:40.993482 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd"] Oct 30 19:58:40.993715 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:40.993694 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" podUID="2547bf26-2d3b-4565-b16c-c37d17d410d7" containerName="istio-proxy" containerID="cri-o://44b7bcfcb36cb8745819e8aed0eb637513b29a0ca49dbf2ca71ce385a7679e31" gracePeriod=30 Oct 30 19:58:41.865810 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:41.865774 5047 patch_prober.go:29] interesting pod/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 19:58:41.866237 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:41.865836 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" podUID="8856e033-793e-4e57-abbf-ae7ae07cbff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 19:58:42.440029 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:42.437171 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:58:42.866436 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:42.866394 5047 patch_prober.go:29] interesting pod/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 19:58:42.866858 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:42.866467 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" podUID="8856e033-793e-4e57-abbf-ae7ae07cbff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 19:58:43.866717 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:43.866681 5047 patch_prober.go:29] interesting pod/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 19:58:43.867218 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:43.866733 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" podUID="8856e033-793e-4e57-abbf-ae7ae07cbff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 19:58:44.866451 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:44.866364 5047 patch_prober.go:29] interesting pod/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 19:58:44.866451 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:44.866431 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" podUID="8856e033-793e-4e57-abbf-ae7ae07cbff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 19:58:45.871383 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:45.871348 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:58:45.872527 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:45.872502 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 19:58:46.128196 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.128134 5047 generic.go:356] "Generic (PLEG): container finished" podID="2547bf26-2d3b-4565-b16c-c37d17d410d7" containerID="44b7bcfcb36cb8745819e8aed0eb637513b29a0ca49dbf2ca71ce385a7679e31" exitCode=0 Oct 30 19:58:46.128321 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.128207 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" event={"ID":"2547bf26-2d3b-4565-b16c-c37d17d410d7","Type":"ContainerDied","Data":"44b7bcfcb36cb8745819e8aed0eb637513b29a0ca49dbf2ca71ce385a7679e31"} Oct 30 19:58:46.128321 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.128270 5047 scope.go:122] "RemoveContainer" containerID="b55a051a5b0d7c5e39a75bf8e4f9b5812b6845acf9cc2d890ed4a9cc885b843e" Oct 30 19:58:46.276839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.276817 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:58:46.355571 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.355541 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-credential-socket\") pod \"2547bf26-2d3b-4565-b16c-c37d17d410d7\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " Oct 30 19:58:46.355761 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.355599 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2547bf26-2d3b-4565-b16c-c37d17d410d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2547bf26-2d3b-4565-b16c-c37d17d410d7-istiod-ca-cert\") pod \"2547bf26-2d3b-4565-b16c-c37d17d410d7\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " Oct 30 19:58:46.355761 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.355673 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-token\" (UniqueName: \"kubernetes.io/projected/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-token\") pod \"2547bf26-2d3b-4565-b16c-c37d17d410d7\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " Oct 30 19:58:46.355761 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.355700 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2547bf26-2d3b-4565-b16c-c37d17d410d7-kube-api-access-67llt\" (UniqueName: \"kubernetes.io/projected/2547bf26-2d3b-4565-b16c-c37d17d410d7-kube-api-access-67llt\") pod \"2547bf26-2d3b-4565-b16c-c37d17d410d7\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " Oct 30 19:58:46.356239 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.355955 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-credential-socket" pod "2547bf26-2d3b-4565-b16c-c37d17d410d7" (UID: "2547bf26-2d3b-4565-b16c-c37d17d410d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:46.356239 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.355993 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-workload-socket\") pod \"2547bf26-2d3b-4565-b16c-c37d17d410d7\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " Oct 30 19:58:46.356239 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.356133 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-data\") pod \"2547bf26-2d3b-4565-b16c-c37d17d410d7\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " Oct 30 19:58:46.356239 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.356194 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-workload-socket" pod "2547bf26-2d3b-4565-b16c-c37d17d410d7" (UID: "2547bf26-2d3b-4565-b16c-c37d17d410d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:46.356239 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.356202 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2547bf26-2d3b-4565-b16c-c37d17d410d7-istiod-ca-cert" pod "2547bf26-2d3b-4565-b16c-c37d17d410d7" (UID: "2547bf26-2d3b-4565-b16c-c37d17d410d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:46.356627 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.356395 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-data" pod "2547bf26-2d3b-4565-b16c-c37d17d410d7" (UID: "2547bf26-2d3b-4565-b16c-c37d17d410d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:46.356627 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.356502 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-envoy\") pod \"2547bf26-2d3b-4565-b16c-c37d17d410d7\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " Oct 30 19:58:46.356755 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.356625 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-podinfo\") pod \"2547bf26-2d3b-4565-b16c-c37d17d410d7\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " Oct 30 19:58:46.357101 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.356661 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-workload-certs\") pod \"2547bf26-2d3b-4565-b16c-c37d17d410d7\" (UID: \"2547bf26-2d3b-4565-b16c-c37d17d410d7\") " Oct 30 19:58:46.357283 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.357249 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-workload-certs" pod "2547bf26-2d3b-4565-b16c-c37d17d410d7" (UID: "2547bf26-2d3b-4565-b16c-c37d17d410d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:46.357843 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.357818 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:46.357949 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.357850 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:46.357949 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.357866 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:46.357949 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.357880 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:46.357949 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.357893 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2547bf26-2d3b-4565-b16c-c37d17d410d7-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:46.358444 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.358420 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2547bf26-2d3b-4565-b16c-c37d17d410d7-kube-api-access-67llt" pod "2547bf26-2d3b-4565-b16c-c37d17d410d7" (UID: "2547bf26-2d3b-4565-b16c-c37d17d410d7"). InnerVolumeSpecName "kube-api-access-67llt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:46.358540 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.358521 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-token" pod "2547bf26-2d3b-4565-b16c-c37d17d410d7" (UID: "2547bf26-2d3b-4565-b16c-c37d17d410d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:46.358997 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.358957 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-envoy" pod "2547bf26-2d3b-4565-b16c-c37d17d410d7" (UID: "2547bf26-2d3b-4565-b16c-c37d17d410d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:46.360050 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.359989 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-podinfo" pod "2547bf26-2d3b-4565-b16c-c37d17d410d7" (UID: "2547bf26-2d3b-4565-b16c-c37d17d410d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:46.458839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.458775 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:46.458839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.458803 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67llt\" (UniqueName: \"kubernetes.io/projected/2547bf26-2d3b-4565-b16c-c37d17d410d7-kube-api-access-67llt\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:46.458839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.458813 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:46.458839 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:46.458822 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2547bf26-2d3b-4565-b16c-c37d17d410d7-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:58:47.136084 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:47.136041 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" event={"ID":"2547bf26-2d3b-4565-b16c-c37d17d410d7","Type":"ContainerDied","Data":"bf9301ce6ed1bba35d3c533ff524ce8b450f2279f2fa69c5c61d1bbb168adc2d"} Oct 30 19:58:47.136455 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:47.136111 5047 scope.go:122] "RemoveContainer" containerID="44b7bcfcb36cb8745819e8aed0eb637513b29a0ca49dbf2ca71ce385a7679e31" Oct 30 19:58:47.136455 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:47.136059 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd" Oct 30 19:58:47.193368 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:47.193340 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd"] Oct 30 19:58:47.203471 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:47.203449 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b36ad-openshift-default-7b78ff587c-lzpkd"] Oct 30 19:58:49.050150 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:49.050073 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2547bf26-2d3b-4565-b16c-c37d17d410d7" path="/var/lib/kubelet/pods/2547bf26-2d3b-4565-b16c-c37d17d410d7/volumes" Oct 30 19:58:50.127257 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:50.127213 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" Oct 30 19:58:51.093953 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:51.093924 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" Oct 30 19:58:55.565727 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:55.565695 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rspbb_8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a/payload-processing/0.log" Oct 30 19:58:55.738689 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:55.738659 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rspbb_8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a/payload-processing/0.log" Oct 30 19:58:55.938374 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:55.938347 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff_0c851817-5127-4d01-91db-5f6bf9e8a062/payload-processing/0.log" Oct 30 19:58:56.135490 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:56.135463 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff_0c851817-5127-4d01-91db-5f6bf9e8a062/payload-processing/0.log" Oct 30 19:58:59.130801 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.130760 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4"] Oct 30 19:58:59.132555 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.132527 5047 cpu_manager.go:414] "containerMap: removing container" podUID="2547bf26-2d3b-4565-b16c-c37d17d410d7" containerName="istio-proxy" Oct 30 19:58:59.132728 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.132712 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2547bf26-2d3b-4565-b16c-c37d17d410d7" containerName="istio-proxy" Oct 30 19:58:59.133134 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.133115 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="2547bf26-2d3b-4565-b16c-c37d17d410d7" containerName="istio-proxy" Oct 30 19:58:59.133276 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.133263 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="2547bf26-2d3b-4565-b16c-c37d17d410d7" containerName="istio-proxy" Oct 30 19:58:59.216138 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.216103 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4"] Oct 30 19:58:59.216542 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.216196 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" Oct 30 19:58:59.220733 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.220711 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz"] Oct 30 19:58:59.226894 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.222789 5047 cpu_manager.go:414] "containerMap: removing container" podUID="2547bf26-2d3b-4565-b16c-c37d17d410d7" containerName="istio-proxy" Oct 30 19:58:59.227177 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.227157 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2547bf26-2d3b-4565-b16c-c37d17d410d7" containerName="istio-proxy" Oct 30 19:58:59.258073 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.239189 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fe3e5e\"" Oct 30 19:58:59.258073 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.239391 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fe3e5e-dockercfg-v27b5\"" Oct 30 19:58:59.264478 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.264443 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3c8991-f71f-4500-b4b7-e25088017a29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4\" (UID: \"aa3c8991-f71f-4500-b4b7-e25088017a29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" Oct 30 19:58:59.264727 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.264694 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pjnj\" (UniqueName: \"kubernetes.io/projected/aa3c8991-f71f-4500-b4b7-e25088017a29-kube-api-access-8pjnj\") pod \"payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4\" (UID: \"aa3c8991-f71f-4500-b4b7-e25088017a29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" Oct 30 19:58:59.271321 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.271297 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz"] Oct 30 19:58:59.271517 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.271500 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" Oct 30 19:58:59.367220 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.367184 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3c8991-f71f-4500-b4b7-e25088017a29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4\" (UID: \"aa3c8991-f71f-4500-b4b7-e25088017a29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" Oct 30 19:58:59.367379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.367229 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pjnj\" (UniqueName: \"kubernetes.io/projected/aa3c8991-f71f-4500-b4b7-e25088017a29-kube-api-access-8pjnj\") pod \"payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4\" (UID: \"aa3c8991-f71f-4500-b4b7-e25088017a29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" Oct 30 19:58:59.367379 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.367291 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rcz8\" (UniqueName: \"kubernetes.io/projected/64a39903-20d2-4c30-88a0-57cf7c222212-kube-api-access-7rcz8\") pod \"payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz\" (UID: \"64a39903-20d2-4c30-88a0-57cf7c222212\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" Oct 30 19:58:59.367507 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.367486 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a39903-20d2-4c30-88a0-57cf7c222212-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz\" (UID: \"64a39903-20d2-4c30-88a0-57cf7c222212\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" Oct 30 19:58:59.367941 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.367919 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3c8991-f71f-4500-b4b7-e25088017a29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4\" (UID: \"aa3c8991-f71f-4500-b4b7-e25088017a29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" Oct 30 19:58:59.385789 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.385727 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pjnj\" (UniqueName: \"kubernetes.io/projected/aa3c8991-f71f-4500-b4b7-e25088017a29-kube-api-access-8pjnj\") pod \"payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4\" (UID: \"aa3c8991-f71f-4500-b4b7-e25088017a29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" Oct 30 19:58:59.469150 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.469121 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a39903-20d2-4c30-88a0-57cf7c222212-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz\" (UID: \"64a39903-20d2-4c30-88a0-57cf7c222212\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" Oct 30 19:58:59.469296 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.469283 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rcz8\" (UniqueName: \"kubernetes.io/projected/64a39903-20d2-4c30-88a0-57cf7c222212-kube-api-access-7rcz8\") pod \"payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz\" (UID: \"64a39903-20d2-4c30-88a0-57cf7c222212\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" Oct 30 19:58:59.469815 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.469793 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a39903-20d2-4c30-88a0-57cf7c222212-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz\" (UID: \"64a39903-20d2-4c30-88a0-57cf7c222212\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" Oct 30 19:58:59.486301 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.486281 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rcz8\" (UniqueName: \"kubernetes.io/projected/64a39903-20d2-4c30-88a0-57cf7c222212-kube-api-access-7rcz8\") pod \"payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz\" (UID: \"64a39903-20d2-4c30-88a0-57cf7c222212\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" Oct 30 19:58:59.575091 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.575069 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" Oct 30 19:58:59.587159 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.587132 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" Oct 30 19:58:59.883616 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.883584 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4"] Oct 30 19:58:59.884641 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:58:59.884619 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3c8991_f71f_4500_b4b7_e25088017a29.slice/crio-69fae77aeecfef553f19914abfc213b75941a0feefda727eba524164c6436cbf WatchSource:0}: Error finding container 69fae77aeecfef553f19914abfc213b75941a0feefda727eba524164c6436cbf: Status 404 returned error can't find the container with id 69fae77aeecfef553f19914abfc213b75941a0feefda727eba524164c6436cbf Oct 30 19:58:59.893911 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:58:59.893873 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz"] Oct 30 19:58:59.899590 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:58:59.899566 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a39903_20d2_4c30_88a0_57cf7c222212.slice/crio-b42247c645426bfcd03e8465dddcc04ca80c28a41c412b742dc4fa4ec5a24078 WatchSource:0}: Error finding container b42247c645426bfcd03e8465dddcc04ca80c28a41c412b742dc4fa4ec5a24078: Status 404 returned error can't find the container with id b42247c645426bfcd03e8465dddcc04ca80c28a41c412b742dc4fa4ec5a24078 Oct 30 19:59:00.267160 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:00.267073 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" event={"ID":"aa3c8991-f71f-4500-b4b7-e25088017a29","Type":"ContainerStarted","Data":"4d26713405d19b3fbaac0a062d7077ed5a4ad6a30c76f2e3f0bff2e394ae8edc"} Oct 30 19:59:00.267160 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:00.267139 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" event={"ID":"aa3c8991-f71f-4500-b4b7-e25088017a29","Type":"ContainerStarted","Data":"69fae77aeecfef553f19914abfc213b75941a0feefda727eba524164c6436cbf"} Oct 30 19:59:00.267642 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:00.267171 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" Oct 30 19:59:00.269144 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:00.269117 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" event={"ID":"64a39903-20d2-4c30-88a0-57cf7c222212","Type":"ContainerStarted","Data":"96a79a1b5b747fe61faaf131b7199e38e1edb9731cb4a930ddd044e314e0811c"} Oct 30 19:59:00.269295 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:00.269158 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" event={"ID":"64a39903-20d2-4c30-88a0-57cf7c222212","Type":"ContainerStarted","Data":"b42247c645426bfcd03e8465dddcc04ca80c28a41c412b742dc4fa4ec5a24078"} Oct 30 19:59:00.269295 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:00.269184 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" Oct 30 19:59:00.300230 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:00.299831 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" podStartSLOduration=1.299815168 podStartE2EDuration="1.299815168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:00.299699936 +0000 UTC m=+3293.374155296" watchObservedRunningTime="2026-10-30 19:59:00.299815168 +0000 UTC m=+3293.374270529" Oct 30 19:59:00.335827 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:00.335774 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" podStartSLOduration=1.33575698 podStartE2EDuration="1.33575698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:00.333579114 +0000 UTC m=+3293.408034474" watchObservedRunningTime="2026-10-30 19:59:00.33575698 +0000 UTC m=+3293.410212345" Oct 30 19:59:07.952877 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:07.952847 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:59:07.957781 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:07.957759 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 19:59:09.602606 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:59:09.602567 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 19:59:11.278399 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:11.278371 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" Oct 30 19:59:11.278832 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:11.278806 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" Oct 30 19:59:15.378141 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:15.378102 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm"] Oct 30 19:59:15.378755 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:15.378358 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" podUID="2615a557-d1fd-4fe2-b23f-2785dd623271" containerName="maas-api" containerID="cri-o://f3835b0b42ee438b352303509f756b2defd071b79de4790bf037fa83a9b5d90f" gracePeriod=30 Oct 30 19:59:15.746835 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:15.746802 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:59:15.973129 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:15.972636 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2615a557-d1fd-4fe2-b23f-2785dd623271-kube-api-access-fq7r7\" (UniqueName: \"kubernetes.io/projected/2615a557-d1fd-4fe2-b23f-2785dd623271-kube-api-access-fq7r7\") pod \"2615a557-d1fd-4fe2-b23f-2785dd623271\" (UID: \"2615a557-d1fd-4fe2-b23f-2785dd623271\") " Oct 30 19:59:15.973129 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:15.972844 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2615a557-d1fd-4fe2-b23f-2785dd623271-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2615a557-d1fd-4fe2-b23f-2785dd623271-metrics-certs\") pod \"2615a557-d1fd-4fe2-b23f-2785dd623271\" (UID: \"2615a557-d1fd-4fe2-b23f-2785dd623271\") " Oct 30 19:59:15.973129 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:15.972915 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2615a557-d1fd-4fe2-b23f-2785dd623271-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2615a557-d1fd-4fe2-b23f-2785dd623271-maas-api-tls\") pod \"2615a557-d1fd-4fe2-b23f-2785dd623271\" (UID: \"2615a557-d1fd-4fe2-b23f-2785dd623271\") " Oct 30 19:59:15.973129 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:15.972989 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2615a557-d1fd-4fe2-b23f-2785dd623271-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2615a557-d1fd-4fe2-b23f-2785dd623271-openshift-service-ca\") pod \"2615a557-d1fd-4fe2-b23f-2785dd623271\" (UID: \"2615a557-d1fd-4fe2-b23f-2785dd623271\") " Oct 30 19:59:15.983380 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:15.973568 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2615a557-d1fd-4fe2-b23f-2785dd623271-openshift-service-ca" pod "2615a557-d1fd-4fe2-b23f-2785dd623271" (UID: "2615a557-d1fd-4fe2-b23f-2785dd623271"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:15.983380 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:15.973778 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2615a557-d1fd-4fe2-b23f-2785dd623271-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:59:15.991050 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:15.989525 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2615a557-d1fd-4fe2-b23f-2785dd623271-metrics-certs" pod "2615a557-d1fd-4fe2-b23f-2785dd623271" (UID: "2615a557-d1fd-4fe2-b23f-2785dd623271"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:15.994591 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:15.994544 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2615a557-d1fd-4fe2-b23f-2785dd623271-kube-api-access-fq7r7" pod "2615a557-d1fd-4fe2-b23f-2785dd623271" (UID: "2615a557-d1fd-4fe2-b23f-2785dd623271"). InnerVolumeSpecName "kube-api-access-fq7r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:15.995124 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:15.995085 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2615a557-d1fd-4fe2-b23f-2785dd623271-maas-api-tls" pod "2615a557-d1fd-4fe2-b23f-2785dd623271" (UID: "2615a557-d1fd-4fe2-b23f-2785dd623271"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:16.075763 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:16.075729 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2615a557-d1fd-4fe2-b23f-2785dd623271-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:59:16.075763 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:16.075762 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq7r7\" (UniqueName: \"kubernetes.io/projected/2615a557-d1fd-4fe2-b23f-2785dd623271-kube-api-access-fq7r7\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:59:16.075763 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:16.075780 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2615a557-d1fd-4fe2-b23f-2785dd623271-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:59:16.444183 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:16.444149 5047 generic.go:356] "Generic (PLEG): container finished" podID="2615a557-d1fd-4fe2-b23f-2785dd623271" containerID="f3835b0b42ee438b352303509f756b2defd071b79de4790bf037fa83a9b5d90f" exitCode=0 Oct 30 19:59:16.444676 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:16.444204 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" Oct 30 19:59:16.444676 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:16.444198 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" event={"ID":"2615a557-d1fd-4fe2-b23f-2785dd623271","Type":"ContainerDied","Data":"f3835b0b42ee438b352303509f756b2defd071b79de4790bf037fa83a9b5d90f"} Oct 30 19:59:16.444676 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:16.444361 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm" event={"ID":"2615a557-d1fd-4fe2-b23f-2785dd623271","Type":"ContainerDied","Data":"b1689c97300df86391f8b9e193a26a1ef37c6f9be99e4967e39aaed9fdc9a740"} Oct 30 19:59:16.444676 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:16.444393 5047 scope.go:122] "RemoveContainer" containerID="f3835b0b42ee438b352303509f756b2defd071b79de4790bf037fa83a9b5d90f" Oct 30 19:59:16.455080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:16.455062 5047 scope.go:122] "RemoveContainer" containerID="f3835b0b42ee438b352303509f756b2defd071b79de4790bf037fa83a9b5d90f" Oct 30 19:59:16.455344 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:59:16.455324 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3835b0b42ee438b352303509f756b2defd071b79de4790bf037fa83a9b5d90f\": container with ID starting with f3835b0b42ee438b352303509f756b2defd071b79de4790bf037fa83a9b5d90f not found: ID does not exist" containerID="f3835b0b42ee438b352303509f756b2defd071b79de4790bf037fa83a9b5d90f" Oct 30 19:59:16.455400 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:16.455352 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3835b0b42ee438b352303509f756b2defd071b79de4790bf037fa83a9b5d90f"} err="failed to get container status \"f3835b0b42ee438b352303509f756b2defd071b79de4790bf037fa83a9b5d90f\": rpc error: code = NotFound desc = could not find container \"f3835b0b42ee438b352303509f756b2defd071b79de4790bf037fa83a9b5d90f\": container with ID starting with f3835b0b42ee438b352303509f756b2defd071b79de4790bf037fa83a9b5d90f not found: ID does not exist" Oct 30 19:59:16.491274 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:16.491249 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm"] Oct 30 19:59:16.501085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:16.501061 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-6e0c55-756568bd-2vhzm"] Oct 30 19:59:17.050504 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:17.050471 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2615a557-d1fd-4fe2-b23f-2785dd623271" path="/var/lib/kubelet/pods/2615a557-d1fd-4fe2-b23f-2785dd623271/volumes" Oct 30 19:59:19.952828 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:19.952786 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq"] Oct 30 19:59:19.953904 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:19.953881 5047 cpu_manager.go:414] "containerMap: removing container" podUID="2615a557-d1fd-4fe2-b23f-2785dd623271" containerName="maas-api" Oct 30 19:59:19.953951 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:19.953910 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2615a557-d1fd-4fe2-b23f-2785dd623271" containerName="maas-api" Oct 30 19:59:19.954157 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:19.954139 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="2615a557-d1fd-4fe2-b23f-2785dd623271" containerName="maas-api" Oct 30 19:59:19.959441 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:19.959421 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 19:59:19.978132 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:19.978110 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq"] Oct 30 19:59:19.983339 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:19.981790 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-e430a0-serving-cert\"" Oct 30 19:59:20.028184 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:20.028149 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce2235f8-00fb-44bc-90f8-6629cfc22510-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq\" (UID: \"ce2235f8-00fb-44bc-90f8-6629cfc22510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 19:59:20.028361 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:20.028259 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrcsx\" (UniqueName: \"kubernetes.io/projected/ce2235f8-00fb-44bc-90f8-6629cfc22510-kube-api-access-nrcsx\") pod \"maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq\" (UID: \"ce2235f8-00fb-44bc-90f8-6629cfc22510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 19:59:20.028442 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:20.028377 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce2235f8-00fb-44bc-90f8-6629cfc22510-metrics-certs\") pod \"maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq\" (UID: \"ce2235f8-00fb-44bc-90f8-6629cfc22510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 19:59:20.028442 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:20.028403 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2235f8-00fb-44bc-90f8-6629cfc22510-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq\" (UID: \"ce2235f8-00fb-44bc-90f8-6629cfc22510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 19:59:20.152218 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:20.152176 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce2235f8-00fb-44bc-90f8-6629cfc22510-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq\" (UID: \"ce2235f8-00fb-44bc-90f8-6629cfc22510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 19:59:20.153622 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:20.153592 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrcsx\" (UniqueName: \"kubernetes.io/projected/ce2235f8-00fb-44bc-90f8-6629cfc22510-kube-api-access-nrcsx\") pod \"maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq\" (UID: \"ce2235f8-00fb-44bc-90f8-6629cfc22510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 19:59:20.158248 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:20.158217 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce2235f8-00fb-44bc-90f8-6629cfc22510-metrics-certs\") pod \"maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq\" (UID: \"ce2235f8-00fb-44bc-90f8-6629cfc22510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 19:59:20.166852 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:20.166811 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2235f8-00fb-44bc-90f8-6629cfc22510-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq\" (UID: \"ce2235f8-00fb-44bc-90f8-6629cfc22510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 19:59:20.178707 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:20.178598 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2235f8-00fb-44bc-90f8-6629cfc22510-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq\" (UID: \"ce2235f8-00fb-44bc-90f8-6629cfc22510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 19:59:20.181605 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:20.181470 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrcsx\" (UniqueName: \"kubernetes.io/projected/ce2235f8-00fb-44bc-90f8-6629cfc22510-kube-api-access-nrcsx\") pod \"maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq\" (UID: \"ce2235f8-00fb-44bc-90f8-6629cfc22510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 19:59:20.186681 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:20.186650 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce2235f8-00fb-44bc-90f8-6629cfc22510-metrics-certs\") pod \"maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq\" (UID: \"ce2235f8-00fb-44bc-90f8-6629cfc22510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 19:59:20.189618 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:20.189589 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce2235f8-00fb-44bc-90f8-6629cfc22510-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq\" (UID: \"ce2235f8-00fb-44bc-90f8-6629cfc22510\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 19:59:20.277606 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:20.277539 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 19:59:20.582448 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:20.582411 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq"] Oct 30 19:59:20.583318 ip-10-0-140-235 kubenswrapper[5047]: W1030 19:59:20.583293 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2235f8_00fb_44bc_90f8_6629cfc22510.slice/crio-ef97b891d640f370c85f05e7604bd68023bf7cfbac46bb760b5bf2e34c88e38f WatchSource:0}: Error finding container ef97b891d640f370c85f05e7604bd68023bf7cfbac46bb760b5bf2e34c88e38f: Status 404 returned error can't find the container with id ef97b891d640f370c85f05e7604bd68023bf7cfbac46bb760b5bf2e34c88e38f Oct 30 19:59:21.481757 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:21.481713 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" event={"ID":"ce2235f8-00fb-44bc-90f8-6629cfc22510","Type":"ContainerStarted","Data":"2ccd7acd3c6787922296acfdf072bf33e68ec34f2d58c294ba41cd7a6de14637"} Oct 30 19:59:21.482263 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:21.481783 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" event={"ID":"ce2235f8-00fb-44bc-90f8-6629cfc22510","Type":"ContainerStarted","Data":"ef97b891d640f370c85f05e7604bd68023bf7cfbac46bb760b5bf2e34c88e38f"} Oct 30 19:59:21.482263 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:21.481809 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 19:59:21.520042 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:21.519976 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" podStartSLOduration=2.213379209 podStartE2EDuration="2.519957797s" totalImagesPullingTime="306.578588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:21.517646075 +0000 UTC m=+3314.592101435" watchObservedRunningTime="2026-10-30 19:59:21.519957797 +0000 UTC m=+3314.594413157" Oct 30 19:59:25.229225 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.229189 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz"] Oct 30 19:59:25.247077 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.246958 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz"] Oct 30 19:59:25.247493 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.247412 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" Oct 30 19:59:25.293307 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.284904 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6e6016\"" Oct 30 19:59:25.293307 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.285500 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6e6016-dockercfg-s68lm\"" Oct 30 19:59:25.314261 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.314209 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mkfp\" (UniqueName: \"kubernetes.io/projected/6092600c-64f3-401b-a6a6-7c322bfbfb28-kube-api-access-5mkfp\") pod \"payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz\" (UID: \"6092600c-64f3-401b-a6a6-7c322bfbfb28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" Oct 30 19:59:25.314463 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.314340 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6092600c-64f3-401b-a6a6-7c322bfbfb28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz\" (UID: \"6092600c-64f3-401b-a6a6-7c322bfbfb28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" Oct 30 19:59:25.370270 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.370238 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm"] Oct 30 19:59:25.374929 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.374905 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" Oct 30 19:59:25.393063 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.393038 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm"] Oct 30 19:59:25.416274 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.416239 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mkfp\" (UniqueName: \"kubernetes.io/projected/6092600c-64f3-401b-a6a6-7c322bfbfb28-kube-api-access-5mkfp\") pod \"payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz\" (UID: \"6092600c-64f3-401b-a6a6-7c322bfbfb28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" Oct 30 19:59:25.416407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.416314 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6092600c-64f3-401b-a6a6-7c322bfbfb28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz\" (UID: \"6092600c-64f3-401b-a6a6-7c322bfbfb28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" Oct 30 19:59:25.416407 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.416364 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96p99\" (UniqueName: \"kubernetes.io/projected/996764e6-cacf-4b77-bc37-8c0323ce29b5-kube-api-access-96p99\") pod \"payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm\" (UID: \"996764e6-cacf-4b77-bc37-8c0323ce29b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" Oct 30 19:59:25.416549 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.416414 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/996764e6-cacf-4b77-bc37-8c0323ce29b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm\" (UID: \"996764e6-cacf-4b77-bc37-8c0323ce29b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" Oct 30 19:59:25.417265 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.417240 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6092600c-64f3-401b-a6a6-7c322bfbfb28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz\" (UID: \"6092600c-64f3-401b-a6a6-7c322bfbfb28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" Oct 30 19:59:25.433354 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.433329 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mkfp\" (UniqueName: \"kubernetes.io/projected/6092600c-64f3-401b-a6a6-7c322bfbfb28-kube-api-access-5mkfp\") pod \"payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz\" (UID: \"6092600c-64f3-401b-a6a6-7c322bfbfb28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" Oct 30 19:59:25.518453 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.518379 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96p99\" (UniqueName: \"kubernetes.io/projected/996764e6-cacf-4b77-bc37-8c0323ce29b5-kube-api-access-96p99\") pod \"payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm\" (UID: \"996764e6-cacf-4b77-bc37-8c0323ce29b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" Oct 30 19:59:25.518599 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.518449 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/996764e6-cacf-4b77-bc37-8c0323ce29b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm\" (UID: \"996764e6-cacf-4b77-bc37-8c0323ce29b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" Oct 30 19:59:25.519206 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.519183 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/996764e6-cacf-4b77-bc37-8c0323ce29b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm\" (UID: \"996764e6-cacf-4b77-bc37-8c0323ce29b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" Oct 30 19:59:25.535443 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.535418 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96p99\" (UniqueName: \"kubernetes.io/projected/996764e6-cacf-4b77-bc37-8c0323ce29b5-kube-api-access-96p99\") pod \"payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm\" (UID: \"996764e6-cacf-4b77-bc37-8c0323ce29b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" Oct 30 19:59:25.606347 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.606315 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" Oct 30 19:59:25.690329 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.690298 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" Oct 30 19:59:25.937990 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:25.923108 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz"] Oct 30 19:59:26.016085 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:26.016060 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm"] Oct 30 19:59:26.522794 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:26.522758 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" event={"ID":"6092600c-64f3-401b-a6a6-7c322bfbfb28","Type":"ContainerStarted","Data":"03f30b64b3a903d3faa9b51d1235baf4f7c13912622a18d369965c36d7d3423e"} Oct 30 19:59:26.523254 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:26.522821 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" event={"ID":"6092600c-64f3-401b-a6a6-7c322bfbfb28","Type":"ContainerStarted","Data":"647f360e6b564676b8962f65b65a0dfe18071ad87255c14199efcc74b6c9f799"} Oct 30 19:59:26.523254 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:26.522853 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" Oct 30 19:59:26.524541 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:26.524515 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" event={"ID":"996764e6-cacf-4b77-bc37-8c0323ce29b5","Type":"ContainerStarted","Data":"7fba70305232e4baef86a07d1800456c736e2500ed70bcaff808cf0999e0da4e"} Oct 30 19:59:26.524688 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:26.524557 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" event={"ID":"996764e6-cacf-4b77-bc37-8c0323ce29b5","Type":"ContainerStarted","Data":"035907c2695f9691bba9f9bb23eb4ad2be54283c00d7e88378ec0883a50c5d55"} Oct 30 19:59:26.524688 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:26.524602 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" Oct 30 19:59:26.561074 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:26.561021 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" podStartSLOduration=1.560989877 podStartE2EDuration="1.560989877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:26.557185635 +0000 UTC m=+3319.631640994" watchObservedRunningTime="2026-10-30 19:59:26.560989877 +0000 UTC m=+3319.635445238" Oct 30 19:59:26.592080 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:26.592026 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" podStartSLOduration=1.592001723 podStartE2EDuration="1.592001723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:26.590592693 +0000 UTC m=+3319.665048051" watchObservedRunningTime="2026-10-30 19:59:26.592001723 +0000 UTC m=+3319.666457119" Oct 30 19:59:27.471830 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:27.471781 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj"] Oct 30 19:59:27.472044 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:27.472019 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" podUID="f9a562ca-d97d-4470-8f56-38ce5ae4b9a4" containerName="payload-processing" containerID="cri-o://0eee5bd4ad4b20bf306f4dce595e7be0ebbb90ad0ef1ad4ce4f0801f7d873f25" gracePeriod=30 Oct 30 19:59:27.494490 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:27.494468 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 19:59:27.908072 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:27.901853 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" Oct 30 19:59:28.000483 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:28.000448 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a562ca-d97d-4470-8f56-38ce5ae4b9a4-kube-api-access-t7kgq\" (UniqueName: \"kubernetes.io/projected/f9a562ca-d97d-4470-8f56-38ce5ae4b9a4-kube-api-access-t7kgq\") pod \"f9a562ca-d97d-4470-8f56-38ce5ae4b9a4\" (UID: \"f9a562ca-d97d-4470-8f56-38ce5ae4b9a4\") " Oct 30 19:59:28.000763 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:28.000741 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9a562ca-d97d-4470-8f56-38ce5ae4b9a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a562ca-d97d-4470-8f56-38ce5ae4b9a4-plugins-config-volume\") pod \"f9a562ca-d97d-4470-8f56-38ce5ae4b9a4\" (UID: \"f9a562ca-d97d-4470-8f56-38ce5ae4b9a4\") " Oct 30 19:59:28.001953 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:28.001926 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a562ca-d97d-4470-8f56-38ce5ae4b9a4-plugins-config-volume" pod "f9a562ca-d97d-4470-8f56-38ce5ae4b9a4" (UID: "f9a562ca-d97d-4470-8f56-38ce5ae4b9a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:28.004868 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:28.004797 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a562ca-d97d-4470-8f56-38ce5ae4b9a4-kube-api-access-t7kgq" pod "f9a562ca-d97d-4470-8f56-38ce5ae4b9a4" (UID: "f9a562ca-d97d-4470-8f56-38ce5ae4b9a4"). InnerVolumeSpecName "kube-api-access-t7kgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:28.102445 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:28.102419 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a562ca-d97d-4470-8f56-38ce5ae4b9a4-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:59:28.102445 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:28.102445 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7kgq\" (UniqueName: \"kubernetes.io/projected/f9a562ca-d97d-4470-8f56-38ce5ae4b9a4-kube-api-access-t7kgq\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 19:59:28.556819 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:28.556726 5047 generic.go:356] "Generic (PLEG): container finished" podID="f9a562ca-d97d-4470-8f56-38ce5ae4b9a4" containerID="0eee5bd4ad4b20bf306f4dce595e7be0ebbb90ad0ef1ad4ce4f0801f7d873f25" exitCode=0 Oct 30 19:59:28.561596 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:28.557063 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" event={"ID":"f9a562ca-d97d-4470-8f56-38ce5ae4b9a4","Type":"ContainerDied","Data":"0eee5bd4ad4b20bf306f4dce595e7be0ebbb90ad0ef1ad4ce4f0801f7d873f25"} Oct 30 19:59:28.561757 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:28.561676 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" event={"ID":"f9a562ca-d97d-4470-8f56-38ce5ae4b9a4","Type":"ContainerDied","Data":"f4bf9264590d6745433c0e74f9e324225351c3b3afda884d1d22b7e52ff24908"} Oct 30 19:59:28.561757 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:28.557174 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj" Oct 30 19:59:28.561757 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:28.561727 5047 scope.go:122] "RemoveContainer" containerID="0eee5bd4ad4b20bf306f4dce595e7be0ebbb90ad0ef1ad4ce4f0801f7d873f25" Oct 30 19:59:28.573181 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:28.573162 5047 scope.go:122] "RemoveContainer" containerID="0eee5bd4ad4b20bf306f4dce595e7be0ebbb90ad0ef1ad4ce4f0801f7d873f25" Oct 30 19:59:28.573470 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:59:28.573444 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eee5bd4ad4b20bf306f4dce595e7be0ebbb90ad0ef1ad4ce4f0801f7d873f25\": container with ID starting with 0eee5bd4ad4b20bf306f4dce595e7be0ebbb90ad0ef1ad4ce4f0801f7d873f25 not found: ID does not exist" containerID="0eee5bd4ad4b20bf306f4dce595e7be0ebbb90ad0ef1ad4ce4f0801f7d873f25" Oct 30 19:59:28.573579 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:28.573474 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eee5bd4ad4b20bf306f4dce595e7be0ebbb90ad0ef1ad4ce4f0801f7d873f25"} err="failed to get container status \"0eee5bd4ad4b20bf306f4dce595e7be0ebbb90ad0ef1ad4ce4f0801f7d873f25\": rpc error: code = NotFound desc = could not find container \"0eee5bd4ad4b20bf306f4dce595e7be0ebbb90ad0ef1ad4ce4f0801f7d873f25\": container with ID starting with 0eee5bd4ad4b20bf306f4dce595e7be0ebbb90ad0ef1ad4ce4f0801f7d873f25 not found: ID does not exist" Oct 30 19:59:28.607624 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:28.607598 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj"] Oct 30 19:59:28.617637 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:28.617614 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-59f9ed-5d66d88544-5jgkj"] Oct 30 19:59:29.050951 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:29.050913 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a562ca-d97d-4470-8f56-38ce5ae4b9a4" path="/var/lib/kubelet/pods/f9a562ca-d97d-4470-8f56-38ce5ae4b9a4/volumes" Oct 30 19:59:37.541693 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:37.541661 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" Oct 30 19:59:37.542155 ip-10-0-140-235 kubenswrapper[5047]: I1030 19:59:37.541793 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" Oct 30 19:59:39.604818 ip-10-0-140-235 kubenswrapper[5047]: E1030 19:59:39.604774 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:00:00.255504 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:00.255457 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd"] Oct 30 20:00:00.256611 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:00.256585 5047 cpu_manager.go:414] "containerMap: removing container" podUID="f9a562ca-d97d-4470-8f56-38ce5ae4b9a4" containerName="payload-processing" Oct 30 20:00:00.256611 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:00.256609 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a562ca-d97d-4470-8f56-38ce5ae4b9a4" containerName="payload-processing" Oct 30 20:00:00.257408 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:00.256959 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a562ca-d97d-4470-8f56-38ce5ae4b9a4" containerName="payload-processing" Oct 30 20:00:00.263236 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:00.263212 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" Oct 30 20:00:00.278574 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:00.278545 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd"] Oct 30 20:00:00.299621 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:00.299588 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxhtp\" (UniqueName: \"kubernetes.io/projected/f10c60fd-c6ab-4d3e-8808-15a141781b2c-kube-api-access-fxhtp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd\" (UID: \"f10c60fd-c6ab-4d3e-8808-15a141781b2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" Oct 30 20:00:00.299752 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:00.299667 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10c60fd-c6ab-4d3e-8808-15a141781b2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd\" (UID: \"f10c60fd-c6ab-4d3e-8808-15a141781b2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" Oct 30 20:00:00.369648 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:00.369613 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:00:00.401389 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:00.401353 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10c60fd-c6ab-4d3e-8808-15a141781b2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd\" (UID: \"f10c60fd-c6ab-4d3e-8808-15a141781b2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" Oct 30 20:00:00.401542 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:00.401525 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxhtp\" (UniqueName: \"kubernetes.io/projected/f10c60fd-c6ab-4d3e-8808-15a141781b2c-kube-api-access-fxhtp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd\" (UID: \"f10c60fd-c6ab-4d3e-8808-15a141781b2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" Oct 30 20:00:00.402001 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:00.401982 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10c60fd-c6ab-4d3e-8808-15a141781b2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd\" (UID: \"f10c60fd-c6ab-4d3e-8808-15a141781b2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" Oct 30 20:00:00.421392 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:00.421362 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxhtp\" (UniqueName: \"kubernetes.io/projected/f10c60fd-c6ab-4d3e-8808-15a141781b2c-kube-api-access-fxhtp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd\" (UID: \"f10c60fd-c6ab-4d3e-8808-15a141781b2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" Oct 30 20:00:00.584670 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:00.584629 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" Oct 30 20:00:00.884623 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:00.884587 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd"] Oct 30 20:00:00.885374 ip-10-0-140-235 kubenswrapper[5047]: W1030 20:00:00.885347 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10c60fd_c6ab_4d3e_8808_15a141781b2c.slice/crio-6034283fb5bafe90344a61b1f80231b09a0f686c42a7af4df2e13ea0615e6811 WatchSource:0}: Error finding container 6034283fb5bafe90344a61b1f80231b09a0f686c42a7af4df2e13ea0615e6811: Status 404 returned error can't find the container with id 6034283fb5bafe90344a61b1f80231b09a0f686c42a7af4df2e13ea0615e6811 Oct 30 20:00:00.917558 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:00.917529 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" event={"ID":"f10c60fd-c6ab-4d3e-8808-15a141781b2c","Type":"ContainerStarted","Data":"6034283fb5bafe90344a61b1f80231b09a0f686c42a7af4df2e13ea0615e6811"} Oct 30 20:00:01.926376 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:01.926341 5047 generic.go:356] "Generic (PLEG): container finished" podID="f10c60fd-c6ab-4d3e-8808-15a141781b2c" containerID="0e46ca0bcff5f3dffb7c1220b086eb0d6e84412b1a4c9749d49ee5b8637867a1" exitCode=60 Oct 30 20:00:01.926840 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:01.926434 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" event={"ID":"f10c60fd-c6ab-4d3e-8808-15a141781b2c","Type":"ContainerDied","Data":"0e46ca0bcff5f3dffb7c1220b086eb0d6e84412b1a4c9749d49ee5b8637867a1"} Oct 30 20:00:01.926840 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:01.926686 5047 scope.go:122] "RemoveContainer" containerID="0e46ca0bcff5f3dffb7c1220b086eb0d6e84412b1a4c9749d49ee5b8637867a1" Oct 30 20:00:02.936318 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:02.936280 5047 generic.go:356] "Generic (PLEG): container finished" podID="f10c60fd-c6ab-4d3e-8808-15a141781b2c" containerID="900e09b70e2260b7ebc924f14b422ce8fd1888de2cad730c9220a16d7f8be8f8" exitCode=60 Oct 30 20:00:02.936815 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:02.936369 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" event={"ID":"f10c60fd-c6ab-4d3e-8808-15a141781b2c","Type":"ContainerDied","Data":"900e09b70e2260b7ebc924f14b422ce8fd1888de2cad730c9220a16d7f8be8f8"} Oct 30 20:00:02.936815 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:02.936441 5047 scope.go:122] "RemoveContainer" containerID="0e46ca0bcff5f3dffb7c1220b086eb0d6e84412b1a4c9749d49ee5b8637867a1" Oct 30 20:00:02.936815 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:02.936710 5047 scope.go:122] "RemoveContainer" containerID="900e09b70e2260b7ebc924f14b422ce8fd1888de2cad730c9220a16d7f8be8f8" Oct 30 20:00:02.936994 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:02.936972 5047 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd_odh-ai-gateway-infra(f10c60fd-c6ab-4d3e-8808-15a141781b2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" podUID="f10c60fd-c6ab-4d3e-8808-15a141781b2c" Oct 30 20:00:03.959448 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:03.959419 5047 scope.go:122] "RemoveContainer" containerID="900e09b70e2260b7ebc924f14b422ce8fd1888de2cad730c9220a16d7f8be8f8" Oct 30 20:00:03.959979 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:03.959695 5047 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd_odh-ai-gateway-infra(f10c60fd-c6ab-4d3e-8808-15a141781b2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" podUID="f10c60fd-c6ab-4d3e-8808-15a141781b2c" Oct 30 20:00:07.950872 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:07.950837 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l"] Oct 30 20:00:07.983841 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:07.983805 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv"] Oct 30 20:00:09.498575 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:09.498542 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z"] Oct 30 20:00:09.499187 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:09.499132 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" podUID="e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4" containerName="maas-api" containerID="cri-o://f430896edb3042f83a988d3c6afabe591e5c531f17f4b61aeb6befe85411fd6c" gracePeriod=30 Oct 30 20:00:09.592818 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:09.592771 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" podUID="e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.103:8443/health\": dial tcp 10.132.0.103:8443: connect: connection refused" Oct 30 20:00:09.601340 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:09.601311 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:00:09.863730 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:09.863698 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 20:00:09.993822 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:09.991728 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-maas-api-tls\") pod \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\" (UID: \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\") " Oct 30 20:00:09.993822 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:09.992035 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-openshift-service-ca\") pod \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\" (UID: \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\") " Oct 30 20:00:09.993822 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:09.992107 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-kube-api-access-tc929\" (UniqueName: \"kubernetes.io/projected/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-kube-api-access-tc929\") pod \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\" (UID: \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\") " Oct 30 20:00:09.993822 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:09.992261 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-metrics-certs\") pod \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\" (UID: \"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4\") " Oct 30 20:00:10.021730 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:10.021667 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-openshift-service-ca" pod "e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4" (UID: "e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:10.021940 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:10.021782 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-metrics-certs" pod "e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4" (UID: "e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:10.073962 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:10.073480 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-kube-api-access-tc929" pod "e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4" (UID: "e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4"). InnerVolumeSpecName "kube-api-access-tc929". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:10.073962 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:10.073558 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-maas-api-tls" pod "e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4" (UID: "e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:10.097186 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:10.096725 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:10.097186 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:10.096756 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc929\" (UniqueName: \"kubernetes.io/projected/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-kube-api-access-tc929\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:10.097186 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:10.096774 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:10.097186 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:10.096789 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:10.118034 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:10.110144 5047 generic.go:356] "Generic (PLEG): container finished" podID="e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4" containerID="f430896edb3042f83a988d3c6afabe591e5c531f17f4b61aeb6befe85411fd6c" exitCode=0 Oct 30 20:00:10.118034 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:10.110277 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" event={"ID":"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4","Type":"ContainerDied","Data":"f430896edb3042f83a988d3c6afabe591e5c531f17f4b61aeb6befe85411fd6c"} Oct 30 20:00:10.118034 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:10.110328 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" event={"ID":"e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4","Type":"ContainerDied","Data":"c17a3bc28ea7afe9126f1d3968c7391bcff2d3820cb56e5dd023bc8108569589"} Oct 30 20:00:10.118034 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:10.110356 5047 scope.go:122] "RemoveContainer" containerID="f430896edb3042f83a988d3c6afabe591e5c531f17f4b61aeb6befe85411fd6c" Oct 30 20:00:10.118034 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:10.111061 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z" Oct 30 20:00:10.138099 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:10.138070 5047 scope.go:122] "RemoveContainer" containerID="f430896edb3042f83a988d3c6afabe591e5c531f17f4b61aeb6befe85411fd6c" Oct 30 20:00:10.138494 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:10.138442 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f430896edb3042f83a988d3c6afabe591e5c531f17f4b61aeb6befe85411fd6c\": container with ID starting with f430896edb3042f83a988d3c6afabe591e5c531f17f4b61aeb6befe85411fd6c not found: ID does not exist" containerID="f430896edb3042f83a988d3c6afabe591e5c531f17f4b61aeb6befe85411fd6c" Oct 30 20:00:10.138593 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:10.138504 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f430896edb3042f83a988d3c6afabe591e5c531f17f4b61aeb6befe85411fd6c"} err="failed to get container status \"f430896edb3042f83a988d3c6afabe591e5c531f17f4b61aeb6befe85411fd6c\": rpc error: code = NotFound desc = could not find container \"f430896edb3042f83a988d3c6afabe591e5c531f17f4b61aeb6befe85411fd6c\": container with ID starting with f430896edb3042f83a988d3c6afabe591e5c531f17f4b61aeb6befe85411fd6c not found: ID does not exist" Oct 30 20:00:10.163979 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:10.163955 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z"] Oct 30 20:00:10.173089 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:10.173068 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a56cb5-5bf5f6d484-v765z"] Oct 30 20:00:11.050946 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:11.050911 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4" path="/var/lib/kubelet/pods/e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4/volumes" Oct 30 20:00:12.951258 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:12.951217 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" podUID="48a34ff4-bd18-4a74-a78f-368c1ac5c651" containerName="payload-processing" containerID="cri-o://3646bfa1c7216c2899f1afa4a0521e7132dee17778d8224e25cbb5ecece7b011" gracePeriod=25 Oct 30 20:00:12.984859 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:12.984817 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" podUID="9566ea28-5afe-4ae9-87d9-c1444aa8aeb3" containerName="payload-pre-processing" containerID="cri-o://18584eedb2d4988aac5058df2aaa5fe162b8922b7a8f479ed07f0214cc533071" gracePeriod=25 Oct 30 20:00:13.138692 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.138658 5047 generic.go:356] "Generic (PLEG): container finished" podID="9566ea28-5afe-4ae9-87d9-c1444aa8aeb3" containerID="18584eedb2d4988aac5058df2aaa5fe162b8922b7a8f479ed07f0214cc533071" exitCode=0 Oct 30 20:00:13.138902 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.138751 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" event={"ID":"9566ea28-5afe-4ae9-87d9-c1444aa8aeb3","Type":"ContainerDied","Data":"18584eedb2d4988aac5058df2aaa5fe162b8922b7a8f479ed07f0214cc533071"} Oct 30 20:00:13.140616 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.140588 5047 generic.go:356] "Generic (PLEG): container finished" podID="48a34ff4-bd18-4a74-a78f-368c1ac5c651" containerID="3646bfa1c7216c2899f1afa4a0521e7132dee17778d8224e25cbb5ecece7b011" exitCode=0 Oct 30 20:00:13.140739 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.140656 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" event={"ID":"48a34ff4-bd18-4a74-a78f-368c1ac5c651","Type":"ContainerDied","Data":"3646bfa1c7216c2899f1afa4a0521e7132dee17778d8224e25cbb5ecece7b011"} Oct 30 20:00:13.283760 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.283729 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" Oct 30 20:00:13.324055 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.322198 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" Oct 30 20:00:13.340059 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.340033 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a34ff4-bd18-4a74-a78f-368c1ac5c651-kube-api-access-gzdt4\" (UniqueName: \"kubernetes.io/projected/48a34ff4-bd18-4a74-a78f-368c1ac5c651-kube-api-access-gzdt4\") pod \"48a34ff4-bd18-4a74-a78f-368c1ac5c651\" (UID: \"48a34ff4-bd18-4a74-a78f-368c1ac5c651\") " Oct 30 20:00:13.340236 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.340221 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a34ff4-bd18-4a74-a78f-368c1ac5c651-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a34ff4-bd18-4a74-a78f-368c1ac5c651-plugins-config-volume\") pod \"48a34ff4-bd18-4a74-a78f-368c1ac5c651\" (UID: \"48a34ff4-bd18-4a74-a78f-368c1ac5c651\") " Oct 30 20:00:13.340770 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.340732 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a34ff4-bd18-4a74-a78f-368c1ac5c651-plugins-config-volume" pod "48a34ff4-bd18-4a74-a78f-368c1ac5c651" (UID: "48a34ff4-bd18-4a74-a78f-368c1ac5c651"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:13.340888 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.340855 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a34ff4-bd18-4a74-a78f-368c1ac5c651-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:13.342508 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.342478 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a34ff4-bd18-4a74-a78f-368c1ac5c651-kube-api-access-gzdt4" pod "48a34ff4-bd18-4a74-a78f-368c1ac5c651" (UID: "48a34ff4-bd18-4a74-a78f-368c1ac5c651"). InnerVolumeSpecName "kube-api-access-gzdt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:13.441518 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.441484 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9566ea28-5afe-4ae9-87d9-c1444aa8aeb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9566ea28-5afe-4ae9-87d9-c1444aa8aeb3-plugins-config-volume\") pod \"9566ea28-5afe-4ae9-87d9-c1444aa8aeb3\" (UID: \"9566ea28-5afe-4ae9-87d9-c1444aa8aeb3\") " Oct 30 20:00:13.441675 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.441662 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9566ea28-5afe-4ae9-87d9-c1444aa8aeb3-kube-api-access-c887m\" (UniqueName: \"kubernetes.io/projected/9566ea28-5afe-4ae9-87d9-c1444aa8aeb3-kube-api-access-c887m\") pod \"9566ea28-5afe-4ae9-87d9-c1444aa8aeb3\" (UID: \"9566ea28-5afe-4ae9-87d9-c1444aa8aeb3\") " Oct 30 20:00:13.441844 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.441820 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9566ea28-5afe-4ae9-87d9-c1444aa8aeb3-plugins-config-volume" pod "9566ea28-5afe-4ae9-87d9-c1444aa8aeb3" (UID: "9566ea28-5afe-4ae9-87d9-c1444aa8aeb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:13.442178 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.442157 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzdt4\" (UniqueName: \"kubernetes.io/projected/48a34ff4-bd18-4a74-a78f-368c1ac5c651-kube-api-access-gzdt4\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:13.442243 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.442182 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9566ea28-5afe-4ae9-87d9-c1444aa8aeb3-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:13.443791 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.443765 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9566ea28-5afe-4ae9-87d9-c1444aa8aeb3-kube-api-access-c887m" pod "9566ea28-5afe-4ae9-87d9-c1444aa8aeb3" (UID: "9566ea28-5afe-4ae9-87d9-c1444aa8aeb3"). InnerVolumeSpecName "kube-api-access-c887m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:13.544050 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:13.543965 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c887m\" (UniqueName: \"kubernetes.io/projected/9566ea28-5afe-4ae9-87d9-c1444aa8aeb3-kube-api-access-c887m\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:14.162671 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:14.162631 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" event={"ID":"9566ea28-5afe-4ae9-87d9-c1444aa8aeb3","Type":"ContainerDied","Data":"5e2a5becdf3121ca5b615598fe2fac5cb3cfdcd781ce51d1a630db3428fb7486"} Oct 30 20:00:14.163079 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:14.162715 5047 scope.go:122] "RemoveContainer" containerID="18584eedb2d4988aac5058df2aaa5fe162b8922b7a8f479ed07f0214cc533071" Oct 30 20:00:14.163079 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:14.162854 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv" Oct 30 20:00:14.200575 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:14.200483 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" event={"ID":"48a34ff4-bd18-4a74-a78f-368c1ac5c651","Type":"ContainerDied","Data":"5414a2b74c0199cf8d0a1f08e5301644c07667921591de8fcbc4b78f1a3ccdd1"} Oct 30 20:00:14.200876 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:14.200775 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l" Oct 30 20:00:14.224807 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:14.224779 5047 scope.go:122] "RemoveContainer" containerID="3646bfa1c7216c2899f1afa4a0521e7132dee17778d8224e25cbb5ecece7b011" Oct 30 20:00:14.251676 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:14.251646 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv"] Oct 30 20:00:14.262075 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:14.262053 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e430a0-6ccd6b946c-bhflv"] Oct 30 20:00:14.290129 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:14.290104 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l"] Oct 30 20:00:14.293366 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:14.293343 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e430a0-74fb694f6-wvx7l"] Oct 30 20:00:15.051981 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:15.051937 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a34ff4-bd18-4a74-a78f-368c1ac5c651" path="/var/lib/kubelet/pods/48a34ff4-bd18-4a74-a78f-368c1ac5c651/volumes" Oct 30 20:00:15.052416 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:15.052400 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9566ea28-5afe-4ae9-87d9-c1444aa8aeb3" path="/var/lib/kubelet/pods/9566ea28-5afe-4ae9-87d9-c1444aa8aeb3/volumes" Oct 30 20:00:17.046647 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:17.046616 5047 scope.go:122] "RemoveContainer" containerID="900e09b70e2260b7ebc924f14b422ce8fd1888de2cad730c9220a16d7f8be8f8" Oct 30 20:00:18.241205 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:18.241124 5047 generic.go:356] "Generic (PLEG): container finished" podID="f10c60fd-c6ab-4d3e-8808-15a141781b2c" containerID="8226b34cdc0a02a61b9c3c61fc37179391e61d214c055c22c426276cb122034d" exitCode=60 Oct 30 20:00:18.241753 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:18.241281 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" event={"ID":"f10c60fd-c6ab-4d3e-8808-15a141781b2c","Type":"ContainerDied","Data":"8226b34cdc0a02a61b9c3c61fc37179391e61d214c055c22c426276cb122034d"} Oct 30 20:00:18.241868 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:18.241844 5047 scope.go:122] "RemoveContainer" containerID="900e09b70e2260b7ebc924f14b422ce8fd1888de2cad730c9220a16d7f8be8f8" Oct 30 20:00:18.242436 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:18.242415 5047 scope.go:122] "RemoveContainer" containerID="8226b34cdc0a02a61b9c3c61fc37179391e61d214c055c22c426276cb122034d" Oct 30 20:00:18.242944 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:18.242916 5047 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd_odh-ai-gateway-infra(f10c60fd-c6ab-4d3e-8808-15a141781b2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" podUID="f10c60fd-c6ab-4d3e-8808-15a141781b2c" Oct 30 20:00:18.719624 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:18.719587 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj"] Oct 30 20:00:18.719930 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:18.719874 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" podUID="70395564-6b94-4b5c-af03-06efe90cb46a" containerName="maas-api" containerID="cri-o://5761efb22616f23972db4b28d1667f2f1f75d323f205f02b711d6b4c458997cf" gracePeriod=30 Oct 30 20:00:18.943049 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:18.942999 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n"] Oct 30 20:00:18.943301 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:18.943271 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" podUID="ccee9fe5-8dfb-4ee8-8e9c-5038485e3783" containerName="payload-processing" containerID="cri-o://17e388310ae3fd2f3f925280c88ed2daf1ea46965ef67c69dbe7351680102ccd" gracePeriod=30 Oct 30 20:00:19.094086 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.090836 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" podUID="ccee9fe5-8dfb-4ee8-8e9c-5038485e3783" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.209:9004: connect: connection refused" Oct 30 20:00:19.105701 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.100460 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 20:00:19.232902 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.232838 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" Oct 30 20:00:19.250456 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.250426 5047 generic.go:356] "Generic (PLEG): container finished" podID="70395564-6b94-4b5c-af03-06efe90cb46a" containerID="5761efb22616f23972db4b28d1667f2f1f75d323f205f02b711d6b4c458997cf" exitCode=0 Oct 30 20:00:19.250805 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.250473 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" Oct 30 20:00:19.250805 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.250474 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" event={"ID":"70395564-6b94-4b5c-af03-06efe90cb46a","Type":"ContainerDied","Data":"5761efb22616f23972db4b28d1667f2f1f75d323f205f02b711d6b4c458997cf"} Oct 30 20:00:19.250805 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.250611 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj" event={"ID":"70395564-6b94-4b5c-af03-06efe90cb46a","Type":"ContainerDied","Data":"3eca61f088f9ff743f2af52e51dbd367624b5fa4a73a5890a0fa70af5c44e45c"} Oct 30 20:00:19.250805 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.250637 5047 scope.go:122] "RemoveContainer" containerID="5761efb22616f23972db4b28d1667f2f1f75d323f205f02b711d6b4c458997cf" Oct 30 20:00:19.259340 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.259316 5047 generic.go:356] "Generic (PLEG): container finished" podID="ccee9fe5-8dfb-4ee8-8e9c-5038485e3783" containerID="17e388310ae3fd2f3f925280c88ed2daf1ea46965ef67c69dbe7351680102ccd" exitCode=0 Oct 30 20:00:19.259506 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.259347 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" event={"ID":"ccee9fe5-8dfb-4ee8-8e9c-5038485e3783","Type":"ContainerDied","Data":"17e388310ae3fd2f3f925280c88ed2daf1ea46965ef67c69dbe7351680102ccd"} Oct 30 20:00:19.259658 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.259543 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" event={"ID":"ccee9fe5-8dfb-4ee8-8e9c-5038485e3783","Type":"ContainerDied","Data":"67511a7855eeadb40af9a8a44077ffd63d1c0065f6c3aeb5bb32b06441c39888"} Oct 30 20:00:19.259658 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.259372 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n" Oct 30 20:00:19.262981 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.262952 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70395564-6b94-4b5c-af03-06efe90cb46a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70395564-6b94-4b5c-af03-06efe90cb46a-maas-api-tls\") pod \"70395564-6b94-4b5c-af03-06efe90cb46a\" (UID: \"70395564-6b94-4b5c-af03-06efe90cb46a\") " Oct 30 20:00:19.263114 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.263038 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70395564-6b94-4b5c-af03-06efe90cb46a-kube-api-access-62xrj\" (UniqueName: \"kubernetes.io/projected/70395564-6b94-4b5c-af03-06efe90cb46a-kube-api-access-62xrj\") pod \"70395564-6b94-4b5c-af03-06efe90cb46a\" (UID: \"70395564-6b94-4b5c-af03-06efe90cb46a\") " Oct 30 20:00:19.263290 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.263269 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70395564-6b94-4b5c-af03-06efe90cb46a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70395564-6b94-4b5c-af03-06efe90cb46a-metrics-certs\") pod \"70395564-6b94-4b5c-af03-06efe90cb46a\" (UID: \"70395564-6b94-4b5c-af03-06efe90cb46a\") " Oct 30 20:00:19.263383 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.263367 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70395564-6b94-4b5c-af03-06efe90cb46a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70395564-6b94-4b5c-af03-06efe90cb46a-openshift-service-ca\") pod \"70395564-6b94-4b5c-af03-06efe90cb46a\" (UID: \"70395564-6b94-4b5c-af03-06efe90cb46a\") " Oct 30 20:00:19.263778 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.263756 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70395564-6b94-4b5c-af03-06efe90cb46a-openshift-service-ca" pod "70395564-6b94-4b5c-af03-06efe90cb46a" (UID: "70395564-6b94-4b5c-af03-06efe90cb46a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:19.264245 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.264225 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70395564-6b94-4b5c-af03-06efe90cb46a-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:19.265207 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.265187 5047 scope.go:122] "RemoveContainer" containerID="5761efb22616f23972db4b28d1667f2f1f75d323f205f02b711d6b4c458997cf" Oct 30 20:00:19.265588 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:19.265565 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5761efb22616f23972db4b28d1667f2f1f75d323f205f02b711d6b4c458997cf\": container with ID starting with 5761efb22616f23972db4b28d1667f2f1f75d323f205f02b711d6b4c458997cf not found: ID does not exist" containerID="5761efb22616f23972db4b28d1667f2f1f75d323f205f02b711d6b4c458997cf" Oct 30 20:00:19.265684 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.265596 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5761efb22616f23972db4b28d1667f2f1f75d323f205f02b711d6b4c458997cf"} err="failed to get container status \"5761efb22616f23972db4b28d1667f2f1f75d323f205f02b711d6b4c458997cf\": rpc error: code = NotFound desc = could not find container \"5761efb22616f23972db4b28d1667f2f1f75d323f205f02b711d6b4c458997cf\": container with ID starting with 5761efb22616f23972db4b28d1667f2f1f75d323f205f02b711d6b4c458997cf not found: ID does not exist" Oct 30 20:00:19.265684 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.265622 5047 scope.go:122] "RemoveContainer" containerID="17e388310ae3fd2f3f925280c88ed2daf1ea46965ef67c69dbe7351680102ccd" Oct 30 20:00:19.265779 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.265719 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70395564-6b94-4b5c-af03-06efe90cb46a-kube-api-access-62xrj" pod "70395564-6b94-4b5c-af03-06efe90cb46a" (UID: "70395564-6b94-4b5c-af03-06efe90cb46a"). InnerVolumeSpecName "kube-api-access-62xrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:19.265830 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.265774 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70395564-6b94-4b5c-af03-06efe90cb46a-metrics-certs" pod "70395564-6b94-4b5c-af03-06efe90cb46a" (UID: "70395564-6b94-4b5c-af03-06efe90cb46a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:19.267230 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.267210 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70395564-6b94-4b5c-af03-06efe90cb46a-maas-api-tls" pod "70395564-6b94-4b5c-af03-06efe90cb46a" (UID: "70395564-6b94-4b5c-af03-06efe90cb46a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:19.278092 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.278066 5047 scope.go:122] "RemoveContainer" containerID="17e388310ae3fd2f3f925280c88ed2daf1ea46965ef67c69dbe7351680102ccd" Oct 30 20:00:19.278678 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:19.278643 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e388310ae3fd2f3f925280c88ed2daf1ea46965ef67c69dbe7351680102ccd\": container with ID starting with 17e388310ae3fd2f3f925280c88ed2daf1ea46965ef67c69dbe7351680102ccd not found: ID does not exist" containerID="17e388310ae3fd2f3f925280c88ed2daf1ea46965ef67c69dbe7351680102ccd" Oct 30 20:00:19.278779 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.278682 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e388310ae3fd2f3f925280c88ed2daf1ea46965ef67c69dbe7351680102ccd"} err="failed to get container status \"17e388310ae3fd2f3f925280c88ed2daf1ea46965ef67c69dbe7351680102ccd\": rpc error: code = NotFound desc = could not find container \"17e388310ae3fd2f3f925280c88ed2daf1ea46965ef67c69dbe7351680102ccd\": container with ID starting with 17e388310ae3fd2f3f925280c88ed2daf1ea46965ef67c69dbe7351680102ccd not found: ID does not exist" Oct 30 20:00:19.291785 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.291759 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd"] Oct 30 20:00:19.369729 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.367163 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccee9fe5-8dfb-4ee8-8e9c-5038485e3783-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccee9fe5-8dfb-4ee8-8e9c-5038485e3783-plugins-config-volume\") pod \"ccee9fe5-8dfb-4ee8-8e9c-5038485e3783\" (UID: \"ccee9fe5-8dfb-4ee8-8e9c-5038485e3783\") " Oct 30 20:00:19.369729 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.367237 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccee9fe5-8dfb-4ee8-8e9c-5038485e3783-kube-api-access-2kd5d\" (UniqueName: \"kubernetes.io/projected/ccee9fe5-8dfb-4ee8-8e9c-5038485e3783-kube-api-access-2kd5d\") pod \"ccee9fe5-8dfb-4ee8-8e9c-5038485e3783\" (UID: \"ccee9fe5-8dfb-4ee8-8e9c-5038485e3783\") " Oct 30 20:00:19.369729 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.368504 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70395564-6b94-4b5c-af03-06efe90cb46a-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:19.369729 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.368530 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70395564-6b94-4b5c-af03-06efe90cb46a-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:19.369729 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.368544 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62xrj\" (UniqueName: \"kubernetes.io/projected/70395564-6b94-4b5c-af03-06efe90cb46a-kube-api-access-62xrj\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:19.369729 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.369567 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccee9fe5-8dfb-4ee8-8e9c-5038485e3783-plugins-config-volume" pod "ccee9fe5-8dfb-4ee8-8e9c-5038485e3783" (UID: "ccee9fe5-8dfb-4ee8-8e9c-5038485e3783"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:19.460060 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.454663 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccee9fe5-8dfb-4ee8-8e9c-5038485e3783-kube-api-access-2kd5d" pod "ccee9fe5-8dfb-4ee8-8e9c-5038485e3783" (UID: "ccee9fe5-8dfb-4ee8-8e9c-5038485e3783"). InnerVolumeSpecName "kube-api-access-2kd5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:19.540080 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.529738 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kd5d\" (UniqueName: \"kubernetes.io/projected/ccee9fe5-8dfb-4ee8-8e9c-5038485e3783-kube-api-access-2kd5d\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:19.540080 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.529774 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccee9fe5-8dfb-4ee8-8e9c-5038485e3783-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:19.612445 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.612414 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" Oct 30 20:00:19.621983 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.621954 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj"] Oct 30 20:00:19.626163 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.626137 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf314f-7965bf7df9-t74rj"] Oct 30 20:00:19.699755 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.699715 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n"] Oct 30 20:00:19.705131 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.705106 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf314f-6dcd4559db-2sc5n"] Oct 30 20:00:19.758655 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.753632 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10c60fd-c6ab-4d3e-8808-15a141781b2c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10c60fd-c6ab-4d3e-8808-15a141781b2c-maas-api-service-ca\") pod \"f10c60fd-c6ab-4d3e-8808-15a141781b2c\" (UID: \"f10c60fd-c6ab-4d3e-8808-15a141781b2c\") " Oct 30 20:00:19.758655 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.753726 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10c60fd-c6ab-4d3e-8808-15a141781b2c-kube-api-access-fxhtp\" (UniqueName: \"kubernetes.io/projected/f10c60fd-c6ab-4d3e-8808-15a141781b2c-kube-api-access-fxhtp\") pod \"f10c60fd-c6ab-4d3e-8808-15a141781b2c\" (UID: \"f10c60fd-c6ab-4d3e-8808-15a141781b2c\") " Oct 30 20:00:19.759098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.759016 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10c60fd-c6ab-4d3e-8808-15a141781b2c-maas-api-service-ca" pod "f10c60fd-c6ab-4d3e-8808-15a141781b2c" (UID: "f10c60fd-c6ab-4d3e-8808-15a141781b2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:19.759217 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.759119 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10c60fd-c6ab-4d3e-8808-15a141781b2c-kube-api-access-fxhtp" pod "f10c60fd-c6ab-4d3e-8808-15a141781b2c" (UID: "f10c60fd-c6ab-4d3e-8808-15a141781b2c"). InnerVolumeSpecName "kube-api-access-fxhtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:19.855993 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.855956 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10c60fd-c6ab-4d3e-8808-15a141781b2c-maas-api-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:19.855993 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:19.855987 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxhtp\" (UniqueName: \"kubernetes.io/projected/f10c60fd-c6ab-4d3e-8808-15a141781b2c-kube-api-access-fxhtp\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:20.269502 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:20.269407 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" Oct 30 20:00:20.269502 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:20.269448 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd" event={"ID":"f10c60fd-c6ab-4d3e-8808-15a141781b2c","Type":"ContainerDied","Data":"6034283fb5bafe90344a61b1f80231b09a0f686c42a7af4df2e13ea0615e6811"} Oct 30 20:00:20.270136 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:20.269517 5047 scope.go:122] "RemoveContainer" containerID="8226b34cdc0a02a61b9c3c61fc37179391e61d214c055c22c426276cb122034d" Oct 30 20:00:20.327564 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:20.327537 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd"] Oct 30 20:00:20.349455 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:20.349429 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9eb76-29889840-khxrd"] Oct 30 20:00:21.054567 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:21.054530 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70395564-6b94-4b5c-af03-06efe90cb46a" path="/var/lib/kubelet/pods/70395564-6b94-4b5c-af03-06efe90cb46a/volumes" Oct 30 20:00:21.055065 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:21.055048 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccee9fe5-8dfb-4ee8-8e9c-5038485e3783" path="/var/lib/kubelet/pods/ccee9fe5-8dfb-4ee8-8e9c-5038485e3783/volumes" Oct 30 20:00:21.055454 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:21.055440 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10c60fd-c6ab-4d3e-8808-15a141781b2c" path="/var/lib/kubelet/pods/f10c60fd-c6ab-4d3e-8808-15a141781b2c/volumes" Oct 30 20:00:26.749890 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:26.749850 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz"] Oct 30 20:00:26.750359 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:26.750252 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" podUID="c4e3fe64-b5fe-453a-ae2b-7dec205c7931" containerName="istio-proxy" containerID="cri-o://3095578aa737bb053919356564b3707ea12c112e6dd8875bc3b09ec918e46151" gracePeriod=30 Oct 30 20:00:26.755237 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:26.755210 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g"] Oct 30 20:00:26.755441 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:26.755406 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" podUID="01aec079-7f44-4c06-98a5-cb46a0efeb3b" containerName="istio-proxy" containerID="cri-o://c5e4b7fa79f26a6f78fee3fae951ca6b47163ba1f6f30d9092f5e6575eda8ec3" gracePeriod=30 Oct 30 20:00:28.592485 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.592439 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t"] Oct 30 20:00:28.593342 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593321 5047 cpu_manager.go:414] "containerMap: removing container" podUID="48a34ff4-bd18-4a74-a78f-368c1ac5c651" containerName="payload-processing" Oct 30 20:00:28.593342 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593343 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a34ff4-bd18-4a74-a78f-368c1ac5c651" containerName="payload-processing" Oct 30 20:00:28.593454 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593358 5047 cpu_manager.go:414] "containerMap: removing container" podUID="ccee9fe5-8dfb-4ee8-8e9c-5038485e3783" containerName="payload-processing" Oct 30 20:00:28.593454 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593363 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccee9fe5-8dfb-4ee8-8e9c-5038485e3783" containerName="payload-processing" Oct 30 20:00:28.593454 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593379 5047 cpu_manager.go:414] "containerMap: removing container" podUID="f10c60fd-c6ab-4d3e-8808-15a141781b2c" containerName="cleanup" Oct 30 20:00:28.593454 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593384 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10c60fd-c6ab-4d3e-8808-15a141781b2c" containerName="cleanup" Oct 30 20:00:28.593454 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593417 5047 cpu_manager.go:414] "containerMap: removing container" podUID="70395564-6b94-4b5c-af03-06efe90cb46a" containerName="maas-api" Oct 30 20:00:28.593454 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593422 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70395564-6b94-4b5c-af03-06efe90cb46a" containerName="maas-api" Oct 30 20:00:28.593454 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593428 5047 cpu_manager.go:414] "containerMap: removing container" podUID="e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4" containerName="maas-api" Oct 30 20:00:28.593454 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593433 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4" containerName="maas-api" Oct 30 20:00:28.593454 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593444 5047 cpu_manager.go:414] "containerMap: removing container" podUID="f10c60fd-c6ab-4d3e-8808-15a141781b2c" containerName="cleanup" Oct 30 20:00:28.593454 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593449 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10c60fd-c6ab-4d3e-8808-15a141781b2c" containerName="cleanup" Oct 30 20:00:28.593454 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593457 5047 cpu_manager.go:414] "containerMap: removing container" podUID="9566ea28-5afe-4ae9-87d9-c1444aa8aeb3" containerName="payload-pre-processing" Oct 30 20:00:28.593795 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593462 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9566ea28-5afe-4ae9-87d9-c1444aa8aeb3" containerName="payload-pre-processing" Oct 30 20:00:28.593795 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593577 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10c60fd-c6ab-4d3e-8808-15a141781b2c" containerName="cleanup" Oct 30 20:00:28.593795 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593586 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="70395564-6b94-4b5c-af03-06efe90cb46a" containerName="maas-api" Oct 30 20:00:28.593795 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593593 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c5dfdf-caff-4da5-9e35-19da8ac2e3f4" containerName="maas-api" Oct 30 20:00:28.593795 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593600 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10c60fd-c6ab-4d3e-8808-15a141781b2c" containerName="cleanup" Oct 30 20:00:28.593795 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593607 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="9566ea28-5afe-4ae9-87d9-c1444aa8aeb3" containerName="payload-pre-processing" Oct 30 20:00:28.593795 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593616 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a34ff4-bd18-4a74-a78f-368c1ac5c651" containerName="payload-processing" Oct 30 20:00:28.593795 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593623 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccee9fe5-8dfb-4ee8-8e9c-5038485e3783" containerName="payload-processing" Oct 30 20:00:28.593795 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.593631 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10c60fd-c6ab-4d3e-8808-15a141781b2c" containerName="cleanup" Oct 30 20:00:28.598885 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.598865 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:00:28.618774 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.618744 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t"] Oct 30 20:00:28.623377 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.623352 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a76aa2-serving-cert\"" Oct 30 20:00:28.685195 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.685161 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b11843-e441-4d53-a386-bc1b63404d00-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t\" (UID: \"52b11843-e441-4d53-a386-bc1b63404d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:00:28.685360 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.685248 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52b11843-e441-4d53-a386-bc1b63404d00-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t\" (UID: \"52b11843-e441-4d53-a386-bc1b63404d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:00:28.685436 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.685392 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b11843-e441-4d53-a386-bc1b63404d00-metrics-certs\") pod \"maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t\" (UID: \"52b11843-e441-4d53-a386-bc1b63404d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:00:28.685563 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.685547 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvpg7\" (UniqueName: \"kubernetes.io/projected/52b11843-e441-4d53-a386-bc1b63404d00-kube-api-access-cvpg7\") pod \"maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t\" (UID: \"52b11843-e441-4d53-a386-bc1b63404d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:00:28.786646 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.786600 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b11843-e441-4d53-a386-bc1b63404d00-metrics-certs\") pod \"maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t\" (UID: \"52b11843-e441-4d53-a386-bc1b63404d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:00:28.786812 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.786719 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvpg7\" (UniqueName: \"kubernetes.io/projected/52b11843-e441-4d53-a386-bc1b63404d00-kube-api-access-cvpg7\") pod \"maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t\" (UID: \"52b11843-e441-4d53-a386-bc1b63404d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:00:28.786923 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.786908 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b11843-e441-4d53-a386-bc1b63404d00-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t\" (UID: \"52b11843-e441-4d53-a386-bc1b63404d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:00:28.786992 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.786936 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52b11843-e441-4d53-a386-bc1b63404d00-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t\" (UID: \"52b11843-e441-4d53-a386-bc1b63404d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:00:28.787607 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.787581 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b11843-e441-4d53-a386-bc1b63404d00-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t\" (UID: \"52b11843-e441-4d53-a386-bc1b63404d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:00:28.789383 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.789359 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b11843-e441-4d53-a386-bc1b63404d00-metrics-certs\") pod \"maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t\" (UID: \"52b11843-e441-4d53-a386-bc1b63404d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:00:28.789508 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.789488 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52b11843-e441-4d53-a386-bc1b63404d00-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t\" (UID: \"52b11843-e441-4d53-a386-bc1b63404d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:00:28.804457 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.804433 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvpg7\" (UniqueName: \"kubernetes.io/projected/52b11843-e441-4d53-a386-bc1b63404d00-kube-api-access-cvpg7\") pod \"maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t\" (UID: \"52b11843-e441-4d53-a386-bc1b63404d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:00:28.914026 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:28.913933 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:00:29.221563 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:29.221524 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t"] Oct 30 20:00:29.222720 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:29.222697 5047 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:00:29.337561 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:29.337523 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" event={"ID":"52b11843-e441-4d53-a386-bc1b63404d00","Type":"ContainerStarted","Data":"22f2fc9e353fa171e99e596261a587599c4d37dbd8d24122e2288b179f77fb9b"} Oct 30 20:00:30.346915 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:30.346875 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" event={"ID":"52b11843-e441-4d53-a386-bc1b63404d00","Type":"ContainerStarted","Data":"7a03210aca63a65f695ee7fad88e85874a53373faeb0611044d40e7c4f3f2400"} Oct 30 20:00:30.347427 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:30.346942 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:00:30.386813 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:30.386759 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" podStartSLOduration=2.10134787 podStartE2EDuration="2.386746298s" totalImagesPullingTime="285.398428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:30.385119926 +0000 UTC m=+3383.459575300" watchObservedRunningTime="2026-10-30 20:00:30.386746298 +0000 UTC m=+3383.461201656" Oct 30 20:00:30.434465 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:30.434434 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z"] Oct 30 20:00:30.434712 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:30.434677 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" podUID="1258562d-1c87-43fe-b887-42591c2460d6" containerName="payload-pre-processing" containerID="cri-o://83f0de5c328693acc831ebcd851bb8d7ec5c9d6d124078a32c1b7c47aac14d31" gracePeriod=30 Oct 30 20:00:30.751688 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:30.751660 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" Oct 30 20:00:30.921191 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:30.921095 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1258562d-1c87-43fe-b887-42591c2460d6-kube-api-access-rrlfm\" (UniqueName: \"kubernetes.io/projected/1258562d-1c87-43fe-b887-42591c2460d6-kube-api-access-rrlfm\") pod \"1258562d-1c87-43fe-b887-42591c2460d6\" (UID: \"1258562d-1c87-43fe-b887-42591c2460d6\") " Oct 30 20:00:30.921372 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:30.921250 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1258562d-1c87-43fe-b887-42591c2460d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1258562d-1c87-43fe-b887-42591c2460d6-plugins-config-volume\") pod \"1258562d-1c87-43fe-b887-42591c2460d6\" (UID: \"1258562d-1c87-43fe-b887-42591c2460d6\") " Oct 30 20:00:30.922453 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:30.922415 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1258562d-1c87-43fe-b887-42591c2460d6-plugins-config-volume" pod "1258562d-1c87-43fe-b887-42591c2460d6" (UID: "1258562d-1c87-43fe-b887-42591c2460d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:30.926804 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:30.924139 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1258562d-1c87-43fe-b887-42591c2460d6-kube-api-access-rrlfm" pod "1258562d-1c87-43fe-b887-42591c2460d6" (UID: "1258562d-1c87-43fe-b887-42591c2460d6"). InnerVolumeSpecName "kube-api-access-rrlfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:31.032212 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:31.032168 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1258562d-1c87-43fe-b887-42591c2460d6-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:31.032212 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:31.032212 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrlfm\" (UniqueName: \"kubernetes.io/projected/1258562d-1c87-43fe-b887-42591c2460d6-kube-api-access-rrlfm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:31.355619 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:31.355584 5047 generic.go:356] "Generic (PLEG): container finished" podID="1258562d-1c87-43fe-b887-42591c2460d6" containerID="83f0de5c328693acc831ebcd851bb8d7ec5c9d6d124078a32c1b7c47aac14d31" exitCode=0 Oct 30 20:00:31.356222 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:31.355644 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" Oct 30 20:00:31.356222 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:31.355666 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" event={"ID":"1258562d-1c87-43fe-b887-42591c2460d6","Type":"ContainerDied","Data":"83f0de5c328693acc831ebcd851bb8d7ec5c9d6d124078a32c1b7c47aac14d31"} Oct 30 20:00:31.356222 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:31.355725 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z" event={"ID":"1258562d-1c87-43fe-b887-42591c2460d6","Type":"ContainerDied","Data":"6b15976360894e6cde1624dc69444601cc7e3954cf5f1797d074d3fc65e665c7"} Oct 30 20:00:31.356222 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:31.355754 5047 scope.go:122] "RemoveContainer" containerID="83f0de5c328693acc831ebcd851bb8d7ec5c9d6d124078a32c1b7c47aac14d31" Oct 30 20:00:31.366780 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:31.366760 5047 scope.go:122] "RemoveContainer" containerID="83f0de5c328693acc831ebcd851bb8d7ec5c9d6d124078a32c1b7c47aac14d31" Oct 30 20:00:31.367071 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:31.367047 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f0de5c328693acc831ebcd851bb8d7ec5c9d6d124078a32c1b7c47aac14d31\": container with ID starting with 83f0de5c328693acc831ebcd851bb8d7ec5c9d6d124078a32c1b7c47aac14d31 not found: ID does not exist" containerID="83f0de5c328693acc831ebcd851bb8d7ec5c9d6d124078a32c1b7c47aac14d31" Oct 30 20:00:31.367121 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:31.367081 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f0de5c328693acc831ebcd851bb8d7ec5c9d6d124078a32c1b7c47aac14d31"} err="failed to get container status \"83f0de5c328693acc831ebcd851bb8d7ec5c9d6d124078a32c1b7c47aac14d31\": rpc error: code = NotFound desc = could not find container \"83f0de5c328693acc831ebcd851bb8d7ec5c9d6d124078a32c1b7c47aac14d31\": container with ID starting with 83f0de5c328693acc831ebcd851bb8d7ec5c9d6d124078a32c1b7c47aac14d31 not found: ID does not exist" Oct 30 20:00:31.415328 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:31.415297 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z"] Oct 30 20:00:31.424678 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:31.424652 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ed2ede-74d4dffc99-xnx5z"] Oct 30 20:00:32.057997 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.057911 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 20:00:32.132239 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.132210 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 20:00:32.154076 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.154046 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-credential-socket\") pod \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " Oct 30 20:00:32.154230 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.154149 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-data\") pod \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " Oct 30 20:00:32.154280 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.154237 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-envoy\") pod \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " Oct 30 20:00:32.154348 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.154331 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istiod-ca-cert\") pod \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " Oct 30 20:00:32.154443 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.154426 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-token\" (UniqueName: \"kubernetes.io/projected/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-token\") pod \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " Oct 30 20:00:32.154481 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.154333 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-credential-socket" pod "c4e3fe64-b5fe-453a-ae2b-7dec205c7931" (UID: "c4e3fe64-b5fe-453a-ae2b-7dec205c7931"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:32.154481 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.154404 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-data" pod "c4e3fe64-b5fe-453a-ae2b-7dec205c7931" (UID: "c4e3fe64-b5fe-453a-ae2b-7dec205c7931"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:32.154593 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.154479 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-podinfo\") pod \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " Oct 30 20:00:32.154593 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.154536 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-kube-api-access-bcqhl\" (UniqueName: \"kubernetes.io/projected/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-kube-api-access-bcqhl\") pod \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " Oct 30 20:00:32.154710 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.154596 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-workload-certs\") pod \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " Oct 30 20:00:32.154710 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.154662 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istiod-ca-cert" pod "c4e3fe64-b5fe-453a-ae2b-7dec205c7931" (UID: "c4e3fe64-b5fe-453a-ae2b-7dec205c7931"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:32.154710 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.154671 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-workload-socket\") pod \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\" (UID: \"c4e3fe64-b5fe-453a-ae2b-7dec205c7931\") " Oct 30 20:00:32.154924 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.154904 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-workload-certs" pod "c4e3fe64-b5fe-453a-ae2b-7dec205c7931" (UID: "c4e3fe64-b5fe-453a-ae2b-7dec205c7931"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:32.155216 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.155147 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-workload-socket" pod "c4e3fe64-b5fe-453a-ae2b-7dec205c7931" (UID: "c4e3fe64-b5fe-453a-ae2b-7dec205c7931"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:32.155373 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.155355 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.155462 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.155377 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.155462 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.155390 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.155462 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.155402 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.155462 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.155415 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.156708 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.156684 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-podinfo" pod "c4e3fe64-b5fe-453a-ae2b-7dec205c7931" (UID: "c4e3fe64-b5fe-453a-ae2b-7dec205c7931"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:00:32.156822 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.156802 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-kube-api-access-bcqhl" pod "c4e3fe64-b5fe-453a-ae2b-7dec205c7931" (UID: "c4e3fe64-b5fe-453a-ae2b-7dec205c7931"). InnerVolumeSpecName "kube-api-access-bcqhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:32.157058 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.157038 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-token" pod "c4e3fe64-b5fe-453a-ae2b-7dec205c7931" (UID: "c4e3fe64-b5fe-453a-ae2b-7dec205c7931"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:32.157117 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.157090 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-envoy" pod "c4e3fe64-b5fe-453a-ae2b-7dec205c7931" (UID: "c4e3fe64-b5fe-453a-ae2b-7dec205c7931"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:32.256416 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.256379 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-token\" (UniqueName: \"kubernetes.io/projected/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-token\") pod \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " Oct 30 20:00:32.256598 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.256440 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-workload-socket\") pod \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " Oct 30 20:00:32.256598 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.256486 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-podinfo\") pod \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " Oct 30 20:00:32.256598 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.256583 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-envoy\") pod \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " Oct 30 20:00:32.256772 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.256636 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istiod-ca-cert\") pod \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " Oct 30 20:00:32.256830 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.256766 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-data\") pod \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " Oct 30 20:00:32.256888 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.256849 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-workload-socket" pod "01aec079-7f44-4c06-98a5-cb46a0efeb3b" (UID: "01aec079-7f44-4c06-98a5-cb46a0efeb3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:32.256888 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.256873 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-workload-certs\") pod \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " Oct 30 20:00:32.257022 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.256963 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01aec079-7f44-4c06-98a5-cb46a0efeb3b-kube-api-access-4dsfp\" (UniqueName: \"kubernetes.io/projected/01aec079-7f44-4c06-98a5-cb46a0efeb3b-kube-api-access-4dsfp\") pod \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " Oct 30 20:00:32.257087 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.257044 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-credential-socket\") pod \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\" (UID: \"01aec079-7f44-4c06-98a5-cb46a0efeb3b\") " Oct 30 20:00:32.257156 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.257081 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-data" pod "01aec079-7f44-4c06-98a5-cb46a0efeb3b" (UID: "01aec079-7f44-4c06-98a5-cb46a0efeb3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:32.257156 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.257095 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-workload-certs" pod "01aec079-7f44-4c06-98a5-cb46a0efeb3b" (UID: "01aec079-7f44-4c06-98a5-cb46a0efeb3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:32.257270 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.257171 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istiod-ca-cert" pod "01aec079-7f44-4c06-98a5-cb46a0efeb3b" (UID: "01aec079-7f44-4c06-98a5-cb46a0efeb3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:32.257382 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.257357 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-credential-socket" pod "01aec079-7f44-4c06-98a5-cb46a0efeb3b" (UID: "01aec079-7f44-4c06-98a5-cb46a0efeb3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:32.257684 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.257661 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.257797 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.257689 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.257797 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.257702 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.257797 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.257716 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.257797 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.257728 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.257797 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.257740 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.257797 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.257755 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcqhl\" (UniqueName: \"kubernetes.io/projected/c4e3fe64-b5fe-453a-ae2b-7dec205c7931-kube-api-access-bcqhl\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.257797 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.257770 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.257797 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.257782 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.258917 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.258898 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-podinfo" pod "01aec079-7f44-4c06-98a5-cb46a0efeb3b" (UID: "01aec079-7f44-4c06-98a5-cb46a0efeb3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:00:32.259179 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.259163 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-token" pod "01aec079-7f44-4c06-98a5-cb46a0efeb3b" (UID: "01aec079-7f44-4c06-98a5-cb46a0efeb3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:32.259498 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.259480 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01aec079-7f44-4c06-98a5-cb46a0efeb3b-kube-api-access-4dsfp" pod "01aec079-7f44-4c06-98a5-cb46a0efeb3b" (UID: "01aec079-7f44-4c06-98a5-cb46a0efeb3b"). InnerVolumeSpecName "kube-api-access-4dsfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:32.259549 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.259494 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-envoy" pod "01aec079-7f44-4c06-98a5-cb46a0efeb3b" (UID: "01aec079-7f44-4c06-98a5-cb46a0efeb3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:32.359191 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.359160 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.359191 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.359190 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.359191 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.359199 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01aec079-7f44-4c06-98a5-cb46a0efeb3b-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.359666 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.359208 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dsfp\" (UniqueName: \"kubernetes.io/projected/01aec079-7f44-4c06-98a5-cb46a0efeb3b-kube-api-access-4dsfp\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:32.366717 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.366685 5047 generic.go:356] "Generic (PLEG): container finished" podID="01aec079-7f44-4c06-98a5-cb46a0efeb3b" containerID="c5e4b7fa79f26a6f78fee3fae951ca6b47163ba1f6f30d9092f5e6575eda8ec3" exitCode=0 Oct 30 20:00:32.366887 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.366744 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" Oct 30 20:00:32.366887 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.366763 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" event={"ID":"01aec079-7f44-4c06-98a5-cb46a0efeb3b","Type":"ContainerDied","Data":"c5e4b7fa79f26a6f78fee3fae951ca6b47163ba1f6f30d9092f5e6575eda8ec3"} Oct 30 20:00:32.366887 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.366815 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g" event={"ID":"01aec079-7f44-4c06-98a5-cb46a0efeb3b","Type":"ContainerDied","Data":"6352d38eb20c1efb1601a8f6d4b34d2c648ebc3dcc8fc8429de3d8ad3a9973a6"} Oct 30 20:00:32.366887 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.366844 5047 scope.go:122] "RemoveContainer" containerID="c5e4b7fa79f26a6f78fee3fae951ca6b47163ba1f6f30d9092f5e6575eda8ec3" Oct 30 20:00:32.368893 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.368864 5047 generic.go:356] "Generic (PLEG): container finished" podID="c4e3fe64-b5fe-453a-ae2b-7dec205c7931" containerID="3095578aa737bb053919356564b3707ea12c112e6dd8875bc3b09ec918e46151" exitCode=0 Oct 30 20:00:32.369064 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.368914 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" Oct 30 20:00:32.369064 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.368919 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" event={"ID":"c4e3fe64-b5fe-453a-ae2b-7dec205c7931","Type":"ContainerDied","Data":"3095578aa737bb053919356564b3707ea12c112e6dd8875bc3b09ec918e46151"} Oct 30 20:00:32.369064 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.368966 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz" event={"ID":"c4e3fe64-b5fe-453a-ae2b-7dec205c7931","Type":"ContainerDied","Data":"d051af7d6508c695cb28214fe3b075796fbd5aa8fdbc7b2099cfc5317ed1721f"} Oct 30 20:00:32.381270 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.381250 5047 scope.go:122] "RemoveContainer" containerID="c5e4b7fa79f26a6f78fee3fae951ca6b47163ba1f6f30d9092f5e6575eda8ec3" Oct 30 20:00:32.381833 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:32.381584 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e4b7fa79f26a6f78fee3fae951ca6b47163ba1f6f30d9092f5e6575eda8ec3\": container with ID starting with c5e4b7fa79f26a6f78fee3fae951ca6b47163ba1f6f30d9092f5e6575eda8ec3 not found: ID does not exist" containerID="c5e4b7fa79f26a6f78fee3fae951ca6b47163ba1f6f30d9092f5e6575eda8ec3" Oct 30 20:00:32.381833 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.381613 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e4b7fa79f26a6f78fee3fae951ca6b47163ba1f6f30d9092f5e6575eda8ec3"} err="failed to get container status \"c5e4b7fa79f26a6f78fee3fae951ca6b47163ba1f6f30d9092f5e6575eda8ec3\": rpc error: code = NotFound desc = could not find container \"c5e4b7fa79f26a6f78fee3fae951ca6b47163ba1f6f30d9092f5e6575eda8ec3\": container with ID starting with c5e4b7fa79f26a6f78fee3fae951ca6b47163ba1f6f30d9092f5e6575eda8ec3 not found: ID does not exist" Oct 30 20:00:32.381833 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.381633 5047 scope.go:122] "RemoveContainer" containerID="3095578aa737bb053919356564b3707ea12c112e6dd8875bc3b09ec918e46151" Oct 30 20:00:32.392537 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.392515 5047 scope.go:122] "RemoveContainer" containerID="1c2ff8d309070abb77eff14bdeabca03ab01bc4f16059298cba917c96dfc8659" Oct 30 20:00:32.403336 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.403315 5047 scope.go:122] "RemoveContainer" containerID="3095578aa737bb053919356564b3707ea12c112e6dd8875bc3b09ec918e46151" Oct 30 20:00:32.403668 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:32.403649 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3095578aa737bb053919356564b3707ea12c112e6dd8875bc3b09ec918e46151\": container with ID starting with 3095578aa737bb053919356564b3707ea12c112e6dd8875bc3b09ec918e46151 not found: ID does not exist" containerID="3095578aa737bb053919356564b3707ea12c112e6dd8875bc3b09ec918e46151" Oct 30 20:00:32.403750 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.403683 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3095578aa737bb053919356564b3707ea12c112e6dd8875bc3b09ec918e46151"} err="failed to get container status \"3095578aa737bb053919356564b3707ea12c112e6dd8875bc3b09ec918e46151\": rpc error: code = NotFound desc = could not find container \"3095578aa737bb053919356564b3707ea12c112e6dd8875bc3b09ec918e46151\": container with ID starting with 3095578aa737bb053919356564b3707ea12c112e6dd8875bc3b09ec918e46151 not found: ID does not exist" Oct 30 20:00:32.403750 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.403708 5047 scope.go:122] "RemoveContainer" containerID="1c2ff8d309070abb77eff14bdeabca03ab01bc4f16059298cba917c96dfc8659" Oct 30 20:00:32.404018 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:32.403988 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2ff8d309070abb77eff14bdeabca03ab01bc4f16059298cba917c96dfc8659\": container with ID starting with 1c2ff8d309070abb77eff14bdeabca03ab01bc4f16059298cba917c96dfc8659 not found: ID does not exist" containerID="1c2ff8d309070abb77eff14bdeabca03ab01bc4f16059298cba917c96dfc8659" Oct 30 20:00:32.404064 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.404045 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2ff8d309070abb77eff14bdeabca03ab01bc4f16059298cba917c96dfc8659"} err="failed to get container status \"1c2ff8d309070abb77eff14bdeabca03ab01bc4f16059298cba917c96dfc8659\": rpc error: code = NotFound desc = could not find container \"1c2ff8d309070abb77eff14bdeabca03ab01bc4f16059298cba917c96dfc8659\": container with ID starting with 1c2ff8d309070abb77eff14bdeabca03ab01bc4f16059298cba917c96dfc8659 not found: ID does not exist" Oct 30 20:00:32.427574 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.427543 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g"] Oct 30 20:00:32.431278 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.431251 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-8pq2g"] Oct 30 20:00:32.459883 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.459849 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz"] Oct 30 20:00:32.463424 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:32.463394 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf314f-openshift-default-c8fd8c9bc-nw2zz"] Oct 30 20:00:33.051303 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:33.051274 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01aec079-7f44-4c06-98a5-cb46a0efeb3b" path="/var/lib/kubelet/pods/01aec079-7f44-4c06-98a5-cb46a0efeb3b/volumes" Oct 30 20:00:33.051698 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:33.051683 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1258562d-1c87-43fe-b887-42591c2460d6" path="/var/lib/kubelet/pods/1258562d-1c87-43fe-b887-42591c2460d6/volumes" Oct 30 20:00:33.052156 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:33.052140 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e3fe64-b5fe-453a-ae2b-7dec205c7931" path="/var/lib/kubelet/pods/c4e3fe64-b5fe-453a-ae2b-7dec205c7931/volumes" Oct 30 20:00:36.361974 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:36.361945 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:00:38.517071 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:38.517029 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph"] Oct 30 20:00:38.517631 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:38.517301 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" podUID="e5345813-fe79-4f6a-a6dc-489770c4e2f6" containerName="istio-proxy" containerID="cri-o://c2c74b902fd8f0d709b166bc2316bc645f5aa914939ce3254da7761dd689e971" gracePeriod=30 Oct 30 20:00:39.602817 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:39.602783 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:00:43.810136 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.810068 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 20:00:43.944269 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.944236 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-envoy\") pod \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " Oct 30 20:00:43.944269 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.944274 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5345813-fe79-4f6a-a6dc-489770c4e2f6-kube-api-access-lg5ck\" (UniqueName: \"kubernetes.io/projected/e5345813-fe79-4f6a-a6dc-489770c4e2f6-kube-api-access-lg5ck\") pod \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " Oct 30 20:00:43.944558 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.944294 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-data\") pod \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " Oct 30 20:00:43.944558 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.944312 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-workload-certs\") pod \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " Oct 30 20:00:43.944558 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.944352 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-token\" (UniqueName: \"kubernetes.io/projected/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-token\") pod \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " Oct 30 20:00:43.944558 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.944447 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istiod-ca-cert\") pod \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " Oct 30 20:00:43.944558 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.944485 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-podinfo\") pod \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " Oct 30 20:00:43.944558 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.944529 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-workload-socket\") pod \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " Oct 30 20:00:43.944892 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.944571 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-data" pod "e5345813-fe79-4f6a-a6dc-489770c4e2f6" (UID: "e5345813-fe79-4f6a-a6dc-489770c4e2f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:43.944892 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.944607 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-credential-socket\") pod \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\" (UID: \"e5345813-fe79-4f6a-a6dc-489770c4e2f6\") " Oct 30 20:00:43.944892 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.944778 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-workload-socket" pod "e5345813-fe79-4f6a-a6dc-489770c4e2f6" (UID: "e5345813-fe79-4f6a-a6dc-489770c4e2f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:43.945246 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.944866 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istiod-ca-cert" pod "e5345813-fe79-4f6a-a6dc-489770c4e2f6" (UID: "e5345813-fe79-4f6a-a6dc-489770c4e2f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:43.945246 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.945239 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:43.945395 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.944941 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-workload-certs" pod "e5345813-fe79-4f6a-a6dc-489770c4e2f6" (UID: "e5345813-fe79-4f6a-a6dc-489770c4e2f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:43.945395 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.945262 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:43.945395 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.945109 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-credential-socket" pod "e5345813-fe79-4f6a-a6dc-489770c4e2f6" (UID: "e5345813-fe79-4f6a-a6dc-489770c4e2f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:43.946804 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.946779 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-envoy" pod "e5345813-fe79-4f6a-a6dc-489770c4e2f6" (UID: "e5345813-fe79-4f6a-a6dc-489770c4e2f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:43.946804 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.946795 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5345813-fe79-4f6a-a6dc-489770c4e2f6-kube-api-access-lg5ck" pod "e5345813-fe79-4f6a-a6dc-489770c4e2f6" (UID: "e5345813-fe79-4f6a-a6dc-489770c4e2f6"). InnerVolumeSpecName "kube-api-access-lg5ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:43.946941 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.946786 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-podinfo" pod "e5345813-fe79-4f6a-a6dc-489770c4e2f6" (UID: "e5345813-fe79-4f6a-a6dc-489770c4e2f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:00:43.947083 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:43.947065 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-token" pod "e5345813-fe79-4f6a-a6dc-489770c4e2f6" (UID: "e5345813-fe79-4f6a-a6dc-489770c4e2f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:44.039797 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.038486 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt"] Oct 30 20:00:44.047129 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.047101 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.047129 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.047128 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg5ck\" (UniqueName: \"kubernetes.io/projected/e5345813-fe79-4f6a-a6dc-489770c4e2f6-kube-api-access-lg5ck\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.047328 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.047138 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.047328 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.047151 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.047328 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.047164 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.047328 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.047175 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5345813-fe79-4f6a-a6dc-489770c4e2f6-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.047328 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.047185 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345813-fe79-4f6a-a6dc-489770c4e2f6-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:44.571099 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.571042 5047 generic.go:356] "Generic (PLEG): container finished" podID="e5345813-fe79-4f6a-a6dc-489770c4e2f6" containerID="c2c74b902fd8f0d709b166bc2316bc645f5aa914939ce3254da7761dd689e971" exitCode=0 Oct 30 20:00:44.571317 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.571285 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" event={"ID":"e5345813-fe79-4f6a-a6dc-489770c4e2f6","Type":"ContainerDied","Data":"c2c74b902fd8f0d709b166bc2316bc645f5aa914939ce3254da7761dd689e971"} Oct 30 20:00:44.571384 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.571337 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" event={"ID":"e5345813-fe79-4f6a-a6dc-489770c4e2f6","Type":"ContainerDied","Data":"5950a5a05acaad8066c61b2353059f2e3cb32bcc727389ac4262e729cfe013df"} Oct 30 20:00:44.571384 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.571365 5047 scope.go:122] "RemoveContainer" containerID="c2c74b902fd8f0d709b166bc2316bc645f5aa914939ce3254da7761dd689e971" Oct 30 20:00:44.571547 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.571532 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph" Oct 30 20:00:44.599655 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.599618 5047 scope.go:122] "RemoveContainer" containerID="b12d72f7f657ff5d176c4fbe6d49412e64d5742ac47634f61704dd58b9b096e6" Oct 30 20:00:44.614678 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.614650 5047 scope.go:122] "RemoveContainer" containerID="c2c74b902fd8f0d709b166bc2316bc645f5aa914939ce3254da7761dd689e971" Oct 30 20:00:44.615024 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:44.614975 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c74b902fd8f0d709b166bc2316bc645f5aa914939ce3254da7761dd689e971\": container with ID starting with c2c74b902fd8f0d709b166bc2316bc645f5aa914939ce3254da7761dd689e971 not found: ID does not exist" containerID="c2c74b902fd8f0d709b166bc2316bc645f5aa914939ce3254da7761dd689e971" Oct 30 20:00:44.615117 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.615035 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c74b902fd8f0d709b166bc2316bc645f5aa914939ce3254da7761dd689e971"} err="failed to get container status \"c2c74b902fd8f0d709b166bc2316bc645f5aa914939ce3254da7761dd689e971\": rpc error: code = NotFound desc = could not find container \"c2c74b902fd8f0d709b166bc2316bc645f5aa914939ce3254da7761dd689e971\": container with ID starting with c2c74b902fd8f0d709b166bc2316bc645f5aa914939ce3254da7761dd689e971 not found: ID does not exist" Oct 30 20:00:44.615117 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.615061 5047 scope.go:122] "RemoveContainer" containerID="b12d72f7f657ff5d176c4fbe6d49412e64d5742ac47634f61704dd58b9b096e6" Oct 30 20:00:44.615437 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:44.615409 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b12d72f7f657ff5d176c4fbe6d49412e64d5742ac47634f61704dd58b9b096e6\": container with ID starting with b12d72f7f657ff5d176c4fbe6d49412e64d5742ac47634f61704dd58b9b096e6 not found: ID does not exist" containerID="b12d72f7f657ff5d176c4fbe6d49412e64d5742ac47634f61704dd58b9b096e6" Oct 30 20:00:44.615494 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.615451 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b12d72f7f657ff5d176c4fbe6d49412e64d5742ac47634f61704dd58b9b096e6"} err="failed to get container status \"b12d72f7f657ff5d176c4fbe6d49412e64d5742ac47634f61704dd58b9b096e6\": rpc error: code = NotFound desc = could not find container \"b12d72f7f657ff5d176c4fbe6d49412e64d5742ac47634f61704dd58b9b096e6\": container with ID starting with b12d72f7f657ff5d176c4fbe6d49412e64d5742ac47634f61704dd58b9b096e6 not found: ID does not exist" Oct 30 20:00:44.633168 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.633142 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph"] Oct 30 20:00:44.642310 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:44.642169 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-f83282-openshift-default-5757fc7b9c-jz6ph"] Oct 30 20:00:45.050609 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:45.050577 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5345813-fe79-4f6a-a6dc-489770c4e2f6" path="/var/lib/kubelet/pods/e5345813-fe79-4f6a-a6dc-489770c4e2f6/volumes" Oct 30 20:00:46.630036 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:46.629988 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm"] Oct 30 20:00:46.630455 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:46.630231 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" podUID="996764e6-cacf-4b77-bc37-8c0323ce29b5" containerName="payload-processing" containerID="cri-o://7fba70305232e4baef86a07d1800456c736e2500ed70bcaff808cf0999e0da4e" gracePeriod=30 Oct 30 20:00:46.666637 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:46.666217 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz"] Oct 30 20:00:46.666637 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:46.666522 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" podUID="6092600c-64f3-401b-a6a6-7c322bfbfb28" containerName="payload-pre-processing" containerID="cri-o://03f30b64b3a903d3faa9b51d1235baf4f7c13912622a18d369965c36d7d3423e" gracePeriod=30 Oct 30 20:00:46.760739 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:46.760706 5047 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996764e6_cacf_4b77_bc37_8c0323ce29b5.slice/crio-conmon-7fba70305232e4baef86a07d1800456c736e2500ed70bcaff808cf0999e0da4e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:00:47.140521 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.140492 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" Oct 30 20:00:47.146141 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.146083 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" Oct 30 20:00:47.281186 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.281149 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6092600c-64f3-401b-a6a6-7c322bfbfb28-kube-api-access-5mkfp\" (UniqueName: \"kubernetes.io/projected/6092600c-64f3-401b-a6a6-7c322bfbfb28-kube-api-access-5mkfp\") pod \"6092600c-64f3-401b-a6a6-7c322bfbfb28\" (UID: \"6092600c-64f3-401b-a6a6-7c322bfbfb28\") " Oct 30 20:00:47.281378 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.281245 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6092600c-64f3-401b-a6a6-7c322bfbfb28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6092600c-64f3-401b-a6a6-7c322bfbfb28-plugins-config-volume\") pod \"6092600c-64f3-401b-a6a6-7c322bfbfb28\" (UID: \"6092600c-64f3-401b-a6a6-7c322bfbfb28\") " Oct 30 20:00:47.281378 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.281310 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996764e6-cacf-4b77-bc37-8c0323ce29b5-kube-api-access-96p99\" (UniqueName: \"kubernetes.io/projected/996764e6-cacf-4b77-bc37-8c0323ce29b5-kube-api-access-96p99\") pod \"996764e6-cacf-4b77-bc37-8c0323ce29b5\" (UID: \"996764e6-cacf-4b77-bc37-8c0323ce29b5\") " Oct 30 20:00:47.281497 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.281378 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/996764e6-cacf-4b77-bc37-8c0323ce29b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/996764e6-cacf-4b77-bc37-8c0323ce29b5-plugins-config-volume\") pod \"996764e6-cacf-4b77-bc37-8c0323ce29b5\" (UID: \"996764e6-cacf-4b77-bc37-8c0323ce29b5\") " Oct 30 20:00:47.281697 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.281668 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6092600c-64f3-401b-a6a6-7c322bfbfb28-plugins-config-volume" pod "6092600c-64f3-401b-a6a6-7c322bfbfb28" (UID: "6092600c-64f3-401b-a6a6-7c322bfbfb28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:47.281761 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.281727 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996764e6-cacf-4b77-bc37-8c0323ce29b5-plugins-config-volume" pod "996764e6-cacf-4b77-bc37-8c0323ce29b5" (UID: "996764e6-cacf-4b77-bc37-8c0323ce29b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:47.281950 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.281935 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6092600c-64f3-401b-a6a6-7c322bfbfb28-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:47.281992 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.281953 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/996764e6-cacf-4b77-bc37-8c0323ce29b5-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:47.283388 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.283362 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996764e6-cacf-4b77-bc37-8c0323ce29b5-kube-api-access-96p99" pod "996764e6-cacf-4b77-bc37-8c0323ce29b5" (UID: "996764e6-cacf-4b77-bc37-8c0323ce29b5"). InnerVolumeSpecName "kube-api-access-96p99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:47.283502 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.283406 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6092600c-64f3-401b-a6a6-7c322bfbfb28-kube-api-access-5mkfp" pod "6092600c-64f3-401b-a6a6-7c322bfbfb28" (UID: "6092600c-64f3-401b-a6a6-7c322bfbfb28"). InnerVolumeSpecName "kube-api-access-5mkfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:47.383681 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.383636 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mkfp\" (UniqueName: \"kubernetes.io/projected/6092600c-64f3-401b-a6a6-7c322bfbfb28-kube-api-access-5mkfp\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:47.383681 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.383677 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96p99\" (UniqueName: \"kubernetes.io/projected/996764e6-cacf-4b77-bc37-8c0323ce29b5-kube-api-access-96p99\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:47.594160 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.594126 5047 generic.go:356] "Generic (PLEG): container finished" podID="6092600c-64f3-401b-a6a6-7c322bfbfb28" containerID="03f30b64b3a903d3faa9b51d1235baf4f7c13912622a18d369965c36d7d3423e" exitCode=0 Oct 30 20:00:47.594332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.594172 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" Oct 30 20:00:47.594332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.594203 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" event={"ID":"6092600c-64f3-401b-a6a6-7c322bfbfb28","Type":"ContainerDied","Data":"03f30b64b3a903d3faa9b51d1235baf4f7c13912622a18d369965c36d7d3423e"} Oct 30 20:00:47.594332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.594259 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz" event={"ID":"6092600c-64f3-401b-a6a6-7c322bfbfb28","Type":"ContainerDied","Data":"647f360e6b564676b8962f65b65a0dfe18071ad87255c14199efcc74b6c9f799"} Oct 30 20:00:47.594332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.594288 5047 scope.go:122] "RemoveContainer" containerID="03f30b64b3a903d3faa9b51d1235baf4f7c13912622a18d369965c36d7d3423e" Oct 30 20:00:47.595825 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.595802 5047 generic.go:356] "Generic (PLEG): container finished" podID="996764e6-cacf-4b77-bc37-8c0323ce29b5" containerID="7fba70305232e4baef86a07d1800456c736e2500ed70bcaff808cf0999e0da4e" exitCode=0 Oct 30 20:00:47.595960 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.595839 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" Oct 30 20:00:47.595960 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.595860 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" event={"ID":"996764e6-cacf-4b77-bc37-8c0323ce29b5","Type":"ContainerDied","Data":"7fba70305232e4baef86a07d1800456c736e2500ed70bcaff808cf0999e0da4e"} Oct 30 20:00:47.595960 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.595894 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm" event={"ID":"996764e6-cacf-4b77-bc37-8c0323ce29b5","Type":"ContainerDied","Data":"035907c2695f9691bba9f9bb23eb4ad2be54283c00d7e88378ec0883a50c5d55"} Oct 30 20:00:47.610062 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.609994 5047 scope.go:122] "RemoveContainer" containerID="03f30b64b3a903d3faa9b51d1235baf4f7c13912622a18d369965c36d7d3423e" Oct 30 20:00:47.610456 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:47.610433 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f30b64b3a903d3faa9b51d1235baf4f7c13912622a18d369965c36d7d3423e\": container with ID starting with 03f30b64b3a903d3faa9b51d1235baf4f7c13912622a18d369965c36d7d3423e not found: ID does not exist" containerID="03f30b64b3a903d3faa9b51d1235baf4f7c13912622a18d369965c36d7d3423e" Oct 30 20:00:47.610550 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.610469 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f30b64b3a903d3faa9b51d1235baf4f7c13912622a18d369965c36d7d3423e"} err="failed to get container status \"03f30b64b3a903d3faa9b51d1235baf4f7c13912622a18d369965c36d7d3423e\": rpc error: code = NotFound desc = could not find container \"03f30b64b3a903d3faa9b51d1235baf4f7c13912622a18d369965c36d7d3423e\": container with ID starting with 03f30b64b3a903d3faa9b51d1235baf4f7c13912622a18d369965c36d7d3423e not found: ID does not exist" Oct 30 20:00:47.610550 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.610493 5047 scope.go:122] "RemoveContainer" containerID="7fba70305232e4baef86a07d1800456c736e2500ed70bcaff808cf0999e0da4e" Oct 30 20:00:47.620393 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.620357 5047 scope.go:122] "RemoveContainer" containerID="7fba70305232e4baef86a07d1800456c736e2500ed70bcaff808cf0999e0da4e" Oct 30 20:00:47.620669 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:47.620649 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fba70305232e4baef86a07d1800456c736e2500ed70bcaff808cf0999e0da4e\": container with ID starting with 7fba70305232e4baef86a07d1800456c736e2500ed70bcaff808cf0999e0da4e not found: ID does not exist" containerID="7fba70305232e4baef86a07d1800456c736e2500ed70bcaff808cf0999e0da4e" Oct 30 20:00:47.620740 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.620677 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fba70305232e4baef86a07d1800456c736e2500ed70bcaff808cf0999e0da4e"} err="failed to get container status \"7fba70305232e4baef86a07d1800456c736e2500ed70bcaff808cf0999e0da4e\": rpc error: code = NotFound desc = could not find container \"7fba70305232e4baef86a07d1800456c736e2500ed70bcaff808cf0999e0da4e\": container with ID starting with 7fba70305232e4baef86a07d1800456c736e2500ed70bcaff808cf0999e0da4e not found: ID does not exist" Oct 30 20:00:47.658467 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.658440 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm"] Oct 30 20:00:47.661462 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.661439 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e6016-7f56db66b8-cqwqm"] Oct 30 20:00:47.689898 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.689868 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz"] Oct 30 20:00:47.697721 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:47.697687 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e6016-78bc765fc5-g59qz"] Oct 30 20:00:49.050157 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:49.050123 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6092600c-64f3-401b-a6a6-7c322bfbfb28" path="/var/lib/kubelet/pods/6092600c-64f3-401b-a6a6-7c322bfbfb28/volumes" Oct 30 20:00:49.050598 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:49.050583 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="996764e6-cacf-4b77-bc37-8c0323ce29b5" path="/var/lib/kubelet/pods/996764e6-cacf-4b77-bc37-8c0323ce29b5/volumes" Oct 30 20:00:49.720000 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:49.719965 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2"] Oct 30 20:00:54.514600 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:54.514564 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn"] Oct 30 20:00:54.515136 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:54.514850 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" podUID="8856e033-793e-4e57-abbf-ae7ae07cbff4" containerName="istio-proxy" containerID="cri-o://5ba2ed8723e740c875fb16073eee4548419ce02f38fba3ea0ea9ed6ae2a83446" gracePeriod=30 Oct 30 20:00:54.721271 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:54.721223 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" podUID="54643a49-f1da-4957-a6d6-81fb67991b2e" containerName="payload-processing" containerID="cri-o://d4f10ce7955b30baf1793c139958d87144f98cd7b50aceeaa1313f688da0cf46" gracePeriod=25 Oct 30 20:00:56.924703 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:56.924660 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" podUID="54643a49-f1da-4957-a6d6-81fb67991b2e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:00:59.063525 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.063484 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" containerID="cri-o://63fc02478fe554df547be72df72c45574f0ffd4e645c3dc9d6bf0b4b4d930f72" gracePeriod=45 Oct 30 20:00:59.334944 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.334918 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 20:00:59.443082 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.443044 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-home\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-home\") pod \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " Oct 30 20:00:59.443266 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.443107 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-model-cache\") pod \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " Oct 30 20:00:59.443356 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.443285 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9325c7d7-408b-44dc-a660-9f15ec74f9e6-kube-api-access-4krv5\" (UniqueName: \"kubernetes.io/projected/9325c7d7-408b-44dc-a660-9f15ec74f9e6-kube-api-access-4krv5\") pod \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " Oct 30 20:00:59.443356 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.443297 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-home" pod "9325c7d7-408b-44dc-a660-9f15ec74f9e6" (UID: "9325c7d7-408b-44dc-a660-9f15ec74f9e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:59.445594 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.443369 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-tmp-dir\") pod \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " Oct 30 20:00:59.445594 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.443424 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9325c7d7-408b-44dc-a660-9f15ec74f9e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/9325c7d7-408b-44dc-a660-9f15ec74f9e6-tls-certs\") pod \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " Oct 30 20:00:59.445594 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.443454 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-dshm\") pod \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\" (UID: \"9325c7d7-408b-44dc-a660-9f15ec74f9e6\") " Oct 30 20:00:59.445594 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.443538 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-model-cache" pod "9325c7d7-408b-44dc-a660-9f15ec74f9e6" (UID: "9325c7d7-408b-44dc-a660-9f15ec74f9e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:59.445594 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.444022 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-tmp-dir" pod "9325c7d7-408b-44dc-a660-9f15ec74f9e6" (UID: "9325c7d7-408b-44dc-a660-9f15ec74f9e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:59.445594 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.444109 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:59.445594 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.444131 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:59.445943 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.445785 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9325c7d7-408b-44dc-a660-9f15ec74f9e6-tls-certs" pod "9325c7d7-408b-44dc-a660-9f15ec74f9e6" (UID: "9325c7d7-408b-44dc-a660-9f15ec74f9e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:59.445943 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.445837 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-dshm" pod "9325c7d7-408b-44dc-a660-9f15ec74f9e6" (UID: "9325c7d7-408b-44dc-a660-9f15ec74f9e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:59.445943 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.445918 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9325c7d7-408b-44dc-a660-9f15ec74f9e6-kube-api-access-4krv5" pod "9325c7d7-408b-44dc-a660-9f15ec74f9e6" (UID: "9325c7d7-408b-44dc-a660-9f15ec74f9e6"). InnerVolumeSpecName "kube-api-access-4krv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:59.545870 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.545831 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:59.545870 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.545874 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9325c7d7-408b-44dc-a660-9f15ec74f9e6-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:59.546104 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.545887 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9325c7d7-408b-44dc-a660-9f15ec74f9e6-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:59.546104 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.545901 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4krv5\" (UniqueName: \"kubernetes.io/projected/9325c7d7-408b-44dc-a660-9f15ec74f9e6-kube-api-access-4krv5\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:59.726889 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.726847 5047 generic.go:356] "Generic (PLEG): container finished" podID="8856e033-793e-4e57-abbf-ae7ae07cbff4" containerID="5ba2ed8723e740c875fb16073eee4548419ce02f38fba3ea0ea9ed6ae2a83446" exitCode=0 Oct 30 20:00:59.727059 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.726887 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" event={"ID":"8856e033-793e-4e57-abbf-ae7ae07cbff4","Type":"ContainerDied","Data":"5ba2ed8723e740c875fb16073eee4548419ce02f38fba3ea0ea9ed6ae2a83446"} Oct 30 20:00:59.727059 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.726964 5047 scope.go:122] "RemoveContainer" containerID="b181b06d3a5acaf3e8e45a05e781d6dc9fb86f6b8931d5432014a39e792bd0ff" Oct 30 20:00:59.728833 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.728800 5047 generic.go:356] "Generic (PLEG): container finished" podID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerID="63fc02478fe554df547be72df72c45574f0ffd4e645c3dc9d6bf0b4b4d930f72" exitCode=0 Oct 30 20:00:59.728951 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.728870 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" Oct 30 20:00:59.729124 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.729076 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" event={"ID":"9325c7d7-408b-44dc-a660-9f15ec74f9e6","Type":"ContainerDied","Data":"63fc02478fe554df547be72df72c45574f0ffd4e645c3dc9d6bf0b4b4d930f72"} Oct 30 20:00:59.729124 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.729115 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt" event={"ID":"9325c7d7-408b-44dc-a660-9f15ec74f9e6","Type":"ContainerDied","Data":"4001b8c88687505fe0ffdf4aa80da59e0bd8a2b39597e053a22dbe108c18696a"} Oct 30 20:00:59.754489 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.754459 5047 scope.go:122] "RemoveContainer" containerID="63fc02478fe554df547be72df72c45574f0ffd4e645c3dc9d6bf0b4b4d930f72" Oct 30 20:00:59.767062 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.767033 5047 scope.go:122] "RemoveContainer" containerID="63fc02478fe554df547be72df72c45574f0ffd4e645c3dc9d6bf0b4b4d930f72" Oct 30 20:00:59.767363 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:00:59.767335 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63fc02478fe554df547be72df72c45574f0ffd4e645c3dc9d6bf0b4b4d930f72\": container with ID starting with 63fc02478fe554df547be72df72c45574f0ffd4e645c3dc9d6bf0b4b4d930f72 not found: ID does not exist" containerID="63fc02478fe554df547be72df72c45574f0ffd4e645c3dc9d6bf0b4b4d930f72" Oct 30 20:00:59.767440 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.767375 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63fc02478fe554df547be72df72c45574f0ffd4e645c3dc9d6bf0b4b4d930f72"} err="failed to get container status \"63fc02478fe554df547be72df72c45574f0ffd4e645c3dc9d6bf0b4b4d930f72\": rpc error: code = NotFound desc = could not find container \"63fc02478fe554df547be72df72c45574f0ffd4e645c3dc9d6bf0b4b4d930f72\": container with ID starting with 63fc02478fe554df547be72df72c45574f0ffd4e645c3dc9d6bf0b4b4d930f72 not found: ID does not exist" Oct 30 20:00:59.776240 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.776207 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt"] Oct 30 20:00:59.784573 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.784545 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5ac614/premium-w1-5ac614-kserve-57fd77fb9f-swqjt"] Oct 30 20:00:59.801404 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.801382 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 20:00:59.850571 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.850539 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-envoy\") pod \"8856e033-793e-4e57-abbf-ae7ae07cbff4\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " Oct 30 20:00:59.850722 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.850637 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-data\") pod \"8856e033-793e-4e57-abbf-ae7ae07cbff4\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " Oct 30 20:00:59.850722 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.850670 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8856e033-793e-4e57-abbf-ae7ae07cbff4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8856e033-793e-4e57-abbf-ae7ae07cbff4-istiod-ca-cert\") pod \"8856e033-793e-4e57-abbf-ae7ae07cbff4\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " Oct 30 20:00:59.850722 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.850695 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-credential-socket\") pod \"8856e033-793e-4e57-abbf-ae7ae07cbff4\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " Oct 30 20:00:59.850907 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.850726 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-workload-socket\") pod \"8856e033-793e-4e57-abbf-ae7ae07cbff4\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " Oct 30 20:00:59.850907 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.850762 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-podinfo\") pod \"8856e033-793e-4e57-abbf-ae7ae07cbff4\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " Oct 30 20:00:59.850907 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.850861 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-token\" (UniqueName: \"kubernetes.io/projected/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-token\") pod \"8856e033-793e-4e57-abbf-ae7ae07cbff4\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " Oct 30 20:00:59.851065 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.850936 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8856e033-793e-4e57-abbf-ae7ae07cbff4-kube-api-access-wsb88\" (UniqueName: \"kubernetes.io/projected/8856e033-793e-4e57-abbf-ae7ae07cbff4-kube-api-access-wsb88\") pod \"8856e033-793e-4e57-abbf-ae7ae07cbff4\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " Oct 30 20:00:59.851065 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.850981 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-workload-certs\") pod \"8856e033-793e-4e57-abbf-ae7ae07cbff4\" (UID: \"8856e033-793e-4e57-abbf-ae7ae07cbff4\") " Oct 30 20:00:59.851455 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.851423 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-data" pod "8856e033-793e-4e57-abbf-ae7ae07cbff4" (UID: "8856e033-793e-4e57-abbf-ae7ae07cbff4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:59.851705 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.851664 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-credential-socket" pod "8856e033-793e-4e57-abbf-ae7ae07cbff4" (UID: "8856e033-793e-4e57-abbf-ae7ae07cbff4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:59.851705 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.851671 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-workload-socket" pod "8856e033-793e-4e57-abbf-ae7ae07cbff4" (UID: "8856e033-793e-4e57-abbf-ae7ae07cbff4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:59.851848 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.851728 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:59.851848 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.851789 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8856e033-793e-4e57-abbf-ae7ae07cbff4-istiod-ca-cert" pod "8856e033-793e-4e57-abbf-ae7ae07cbff4" (UID: "8856e033-793e-4e57-abbf-ae7ae07cbff4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:59.852177 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.852153 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-workload-certs" pod "8856e033-793e-4e57-abbf-ae7ae07cbff4" (UID: "8856e033-793e-4e57-abbf-ae7ae07cbff4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:59.854239 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.854214 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-envoy" pod "8856e033-793e-4e57-abbf-ae7ae07cbff4" (UID: "8856e033-793e-4e57-abbf-ae7ae07cbff4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:59.854801 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.854771 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-token" pod "8856e033-793e-4e57-abbf-ae7ae07cbff4" (UID: "8856e033-793e-4e57-abbf-ae7ae07cbff4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:59.854890 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.854871 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-podinfo" pod "8856e033-793e-4e57-abbf-ae7ae07cbff4" (UID: "8856e033-793e-4e57-abbf-ae7ae07cbff4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:00:59.855226 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.855205 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8856e033-793e-4e57-abbf-ae7ae07cbff4-kube-api-access-wsb88" pod "8856e033-793e-4e57-abbf-ae7ae07cbff4" (UID: "8856e033-793e-4e57-abbf-ae7ae07cbff4"). InnerVolumeSpecName "kube-api-access-wsb88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:59.953266 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.953182 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:59.953266 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.953209 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8856e033-793e-4e57-abbf-ae7ae07cbff4-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:59.953266 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.953222 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:59.953266 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.953231 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:59.953266 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.953240 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:59.953266 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.953249 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8856e033-793e-4e57-abbf-ae7ae07cbff4-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:59.953266 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.953257 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsb88\" (UniqueName: \"kubernetes.io/projected/8856e033-793e-4e57-abbf-ae7ae07cbff4-kube-api-access-wsb88\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:00:59.953266 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:00:59.953266 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8856e033-793e-4e57-abbf-ae7ae07cbff4-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:00.737873 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:00.737838 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" Oct 30 20:01:00.737873 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:00.737852 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn" event={"ID":"8856e033-793e-4e57-abbf-ae7ae07cbff4","Type":"ContainerDied","Data":"ab604bffa1f561377f285b1c1082fff8b1bda62fef8267886e952177e0ebe2fd"} Oct 30 20:01:00.738463 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:00.737918 5047 scope.go:122] "RemoveContainer" containerID="5ba2ed8723e740c875fb16073eee4548419ce02f38fba3ea0ea9ed6ae2a83446" Oct 30 20:01:00.794457 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:00.794432 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn"] Oct 30 20:01:00.804290 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:00.804266 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6e6016-openshift-default-5bd646478f-s4zwn"] Oct 30 20:01:01.056844 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:01.056753 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8856e033-793e-4e57-abbf-ae7ae07cbff4" path="/var/lib/kubelet/pods/8856e033-793e-4e57-abbf-ae7ae07cbff4/volumes" Oct 30 20:01:01.057290 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:01.057272 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" path="/var/lib/kubelet/pods/9325c7d7-408b-44dc-a660-9f15ec74f9e6/volumes" Oct 30 20:01:06.924823 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:06.924781 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" podUID="54643a49-f1da-4957-a6d6-81fb67991b2e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:01:09.603160 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:01:09.603117 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:01:12.856082 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:12.856045 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rspbb_8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a/payload-processing/0.log" Oct 30 20:01:14.851805 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:14.851772 5047 generic.go:356] "Generic (PLEG): container finished" podID="54643a49-f1da-4957-a6d6-81fb67991b2e" containerID="d4f10ce7955b30baf1793c139958d87144f98cd7b50aceeaa1313f688da0cf46" exitCode=0 Oct 30 20:01:14.852247 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:14.851825 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" event={"ID":"54643a49-f1da-4957-a6d6-81fb67991b2e","Type":"ContainerDied","Data":"d4f10ce7955b30baf1793c139958d87144f98cd7b50aceeaa1313f688da0cf46"} Oct 30 20:01:15.075773 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:15.075742 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" Oct 30 20:01:15.179408 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:15.179370 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54643a49-f1da-4957-a6d6-81fb67991b2e-kube-api-access-fl8cw\" (UniqueName: \"kubernetes.io/projected/54643a49-f1da-4957-a6d6-81fb67991b2e-kube-api-access-fl8cw\") pod \"54643a49-f1da-4957-a6d6-81fb67991b2e\" (UID: \"54643a49-f1da-4957-a6d6-81fb67991b2e\") " Oct 30 20:01:15.179598 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:15.179542 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54643a49-f1da-4957-a6d6-81fb67991b2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54643a49-f1da-4957-a6d6-81fb67991b2e-plugins-config-volume\") pod \"54643a49-f1da-4957-a6d6-81fb67991b2e\" (UID: \"54643a49-f1da-4957-a6d6-81fb67991b2e\") " Oct 30 20:01:15.179920 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:15.179892 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54643a49-f1da-4957-a6d6-81fb67991b2e-plugins-config-volume" pod "54643a49-f1da-4957-a6d6-81fb67991b2e" (UID: "54643a49-f1da-4957-a6d6-81fb67991b2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:15.180113 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:15.180000 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54643a49-f1da-4957-a6d6-81fb67991b2e-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:15.181741 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:15.181720 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54643a49-f1da-4957-a6d6-81fb67991b2e-kube-api-access-fl8cw" pod "54643a49-f1da-4957-a6d6-81fb67991b2e" (UID: "54643a49-f1da-4957-a6d6-81fb67991b2e"). InnerVolumeSpecName "kube-api-access-fl8cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:15.281551 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:15.281512 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl8cw\" (UniqueName: \"kubernetes.io/projected/54643a49-f1da-4957-a6d6-81fb67991b2e-kube-api-access-fl8cw\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:15.859580 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:15.859543 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" Oct 30 20:01:15.859580 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:15.859559 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2" event={"ID":"54643a49-f1da-4957-a6d6-81fb67991b2e","Type":"ContainerDied","Data":"24446ee291af9dd6296c505a9a6dceea92c871fb629b71e76c92e911c2ba5d86"} Oct 30 20:01:15.860224 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:15.859629 5047 scope.go:122] "RemoveContainer" containerID="d4f10ce7955b30baf1793c139958d87144f98cd7b50aceeaa1313f688da0cf46" Oct 30 20:01:15.904539 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:15.904490 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2"] Oct 30 20:01:15.908348 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:15.908310 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5ac614-67c554ff88-6wjq2"] Oct 30 20:01:17.051382 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:17.051342 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54643a49-f1da-4957-a6d6-81fb67991b2e" path="/var/lib/kubelet/pods/54643a49-f1da-4957-a6d6-81fb67991b2e/volumes" Oct 30 20:01:22.006692 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.006655 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf"] Oct 30 20:01:22.007169 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.006862 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" podUID="874ca58f-1ef6-4395-bc98-892403898cf0" containerName="maas-api" containerID="cri-o://83513bec1c8b847ef7b053866c76211584331c4be1f53958c8365bc953ac2529" gracePeriod=30 Oct 30 20:01:22.497114 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.493240 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 20:01:22.606120 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.604852 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874ca58f-1ef6-4395-bc98-892403898cf0-kube-api-access-fst6n\" (UniqueName: \"kubernetes.io/projected/874ca58f-1ef6-4395-bc98-892403898cf0-kube-api-access-fst6n\") pod \"874ca58f-1ef6-4395-bc98-892403898cf0\" (UID: \"874ca58f-1ef6-4395-bc98-892403898cf0\") " Oct 30 20:01:22.606120 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.604964 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/874ca58f-1ef6-4395-bc98-892403898cf0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/874ca58f-1ef6-4395-bc98-892403898cf0-openshift-service-ca\") pod \"874ca58f-1ef6-4395-bc98-892403898cf0\" (UID: \"874ca58f-1ef6-4395-bc98-892403898cf0\") " Oct 30 20:01:22.606120 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.605038 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/874ca58f-1ef6-4395-bc98-892403898cf0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/874ca58f-1ef6-4395-bc98-892403898cf0-maas-api-tls\") pod \"874ca58f-1ef6-4395-bc98-892403898cf0\" (UID: \"874ca58f-1ef6-4395-bc98-892403898cf0\") " Oct 30 20:01:22.606120 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.605253 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/874ca58f-1ef6-4395-bc98-892403898cf0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/874ca58f-1ef6-4395-bc98-892403898cf0-metrics-certs\") pod \"874ca58f-1ef6-4395-bc98-892403898cf0\" (UID: \"874ca58f-1ef6-4395-bc98-892403898cf0\") " Oct 30 20:01:22.608329 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.608074 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874ca58f-1ef6-4395-bc98-892403898cf0-openshift-service-ca" pod "874ca58f-1ef6-4395-bc98-892403898cf0" (UID: "874ca58f-1ef6-4395-bc98-892403898cf0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:22.610173 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.610137 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/874ca58f-1ef6-4395-bc98-892403898cf0-metrics-certs" pod "874ca58f-1ef6-4395-bc98-892403898cf0" (UID: "874ca58f-1ef6-4395-bc98-892403898cf0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:22.615183 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.615132 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874ca58f-1ef6-4395-bc98-892403898cf0-kube-api-access-fst6n" pod "874ca58f-1ef6-4395-bc98-892403898cf0" (UID: "874ca58f-1ef6-4395-bc98-892403898cf0"). InnerVolumeSpecName "kube-api-access-fst6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:22.615342 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.615228 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/874ca58f-1ef6-4395-bc98-892403898cf0-maas-api-tls" pod "874ca58f-1ef6-4395-bc98-892403898cf0" (UID: "874ca58f-1ef6-4395-bc98-892403898cf0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:22.709845 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.708498 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/874ca58f-1ef6-4395-bc98-892403898cf0-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:22.709845 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.708559 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fst6n\" (UniqueName: \"kubernetes.io/projected/874ca58f-1ef6-4395-bc98-892403898cf0-kube-api-access-fst6n\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:22.709845 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.708575 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/874ca58f-1ef6-4395-bc98-892403898cf0-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:22.709845 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.708592 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/874ca58f-1ef6-4395-bc98-892403898cf0-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:22.941366 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.941277 5047 generic.go:356] "Generic (PLEG): container finished" podID="874ca58f-1ef6-4395-bc98-892403898cf0" containerID="83513bec1c8b847ef7b053866c76211584331c4be1f53958c8365bc953ac2529" exitCode=0 Oct 30 20:01:22.941366 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.941341 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" Oct 30 20:01:22.941366 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.941355 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" event={"ID":"874ca58f-1ef6-4395-bc98-892403898cf0","Type":"ContainerDied","Data":"83513bec1c8b847ef7b053866c76211584331c4be1f53958c8365bc953ac2529"} Oct 30 20:01:22.941608 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.941434 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf" event={"ID":"874ca58f-1ef6-4395-bc98-892403898cf0","Type":"ContainerDied","Data":"fba2e884ec08862031196962ca95e9cb9694af541e4529970f54736f0ebe41a1"} Oct 30 20:01:22.941608 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.941456 5047 scope.go:122] "RemoveContainer" containerID="83513bec1c8b847ef7b053866c76211584331c4be1f53958c8365bc953ac2529" Oct 30 20:01:22.953069 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.953047 5047 scope.go:122] "RemoveContainer" containerID="83513bec1c8b847ef7b053866c76211584331c4be1f53958c8365bc953ac2529" Oct 30 20:01:22.953417 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:01:22.953384 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83513bec1c8b847ef7b053866c76211584331c4be1f53958c8365bc953ac2529\": container with ID starting with 83513bec1c8b847ef7b053866c76211584331c4be1f53958c8365bc953ac2529 not found: ID does not exist" containerID="83513bec1c8b847ef7b053866c76211584331c4be1f53958c8365bc953ac2529" Oct 30 20:01:22.953527 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.953424 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83513bec1c8b847ef7b053866c76211584331c4be1f53958c8365bc953ac2529"} err="failed to get container status \"83513bec1c8b847ef7b053866c76211584331c4be1f53958c8365bc953ac2529\": rpc error: code = NotFound desc = could not find container \"83513bec1c8b847ef7b053866c76211584331c4be1f53958c8365bc953ac2529\": container with ID starting with 83513bec1c8b847ef7b053866c76211584331c4be1f53958c8365bc953ac2529 not found: ID does not exist" Oct 30 20:01:22.991067 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:22.990257 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf"] Oct 30 20:01:23.002942 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:23.002906 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cd4bde-fc496bc5f-lhdcf"] Oct 30 20:01:23.052322 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:23.052284 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="874ca58f-1ef6-4395-bc98-892403898cf0" path="/var/lib/kubelet/pods/874ca58f-1ef6-4395-bc98-892403898cf0/volumes" Oct 30 20:01:23.081294 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:23.081267 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rspbb_8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a/payload-processing/0.log" Oct 30 20:01:34.023342 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:34.023306 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6"] Oct 30 20:01:34.023965 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:34.023568 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" podUID="e45e50f1-ccb5-4cdc-bd7b-75d85074b002" containerName="maas-api" containerID="cri-o://593f43a62fb8913eb5a0146559b68fce4aa948e2892e537b566a58156dfe9cf9" gracePeriod=30 Oct 30 20:01:34.361172 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:34.361142 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 20:01:34.537156 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:34.534564 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-maas-api-tls\") pod \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\" (UID: \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\") " Oct 30 20:01:34.539305 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:34.539269 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-metrics-certs\") pod \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\" (UID: \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\") " Oct 30 20:01:34.539482 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:34.539338 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-kube-api-access-fjq6p\" (UniqueName: \"kubernetes.io/projected/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-kube-api-access-fjq6p\") pod \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\" (UID: \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\") " Oct 30 20:01:34.539545 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:34.539501 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-openshift-service-ca\") pod \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\" (UID: \"e45e50f1-ccb5-4cdc-bd7b-75d85074b002\") " Oct 30 20:01:34.541629 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:34.541585 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-openshift-service-ca" pod "e45e50f1-ccb5-4cdc-bd7b-75d85074b002" (UID: "e45e50f1-ccb5-4cdc-bd7b-75d85074b002"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:34.550213 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:34.550167 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-maas-api-tls" pod "e45e50f1-ccb5-4cdc-bd7b-75d85074b002" (UID: "e45e50f1-ccb5-4cdc-bd7b-75d85074b002"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:34.551467 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:34.551434 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-metrics-certs" pod "e45e50f1-ccb5-4cdc-bd7b-75d85074b002" (UID: "e45e50f1-ccb5-4cdc-bd7b-75d85074b002"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:34.552914 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:34.552884 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-kube-api-access-fjq6p" pod "e45e50f1-ccb5-4cdc-bd7b-75d85074b002" (UID: "e45e50f1-ccb5-4cdc-bd7b-75d85074b002"). InnerVolumeSpecName "kube-api-access-fjq6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:34.641024 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:34.640972 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:34.641024 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:34.641026 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:34.641024 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:34.641037 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:34.641302 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:34.641046 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjq6p\" (UniqueName: \"kubernetes.io/projected/e45e50f1-ccb5-4cdc-bd7b-75d85074b002-kube-api-access-fjq6p\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:35.030523 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:35.030431 5047 generic.go:356] "Generic (PLEG): container finished" podID="e45e50f1-ccb5-4cdc-bd7b-75d85074b002" containerID="593f43a62fb8913eb5a0146559b68fce4aa948e2892e537b566a58156dfe9cf9" exitCode=0 Oct 30 20:01:35.030523 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:35.030474 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" event={"ID":"e45e50f1-ccb5-4cdc-bd7b-75d85074b002","Type":"ContainerDied","Data":"593f43a62fb8913eb5a0146559b68fce4aa948e2892e537b566a58156dfe9cf9"} Oct 30 20:01:35.030523 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:35.030483 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" Oct 30 20:01:35.030523 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:35.030517 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6" event={"ID":"e45e50f1-ccb5-4cdc-bd7b-75d85074b002","Type":"ContainerDied","Data":"98c37ea3c24643f9c001bc91b6af9ed70ac0ea68eff1cc213c2cef2d3c2d6605"} Oct 30 20:01:35.031151 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:35.030539 5047 scope.go:122] "RemoveContainer" containerID="593f43a62fb8913eb5a0146559b68fce4aa948e2892e537b566a58156dfe9cf9" Oct 30 20:01:35.042972 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:35.042946 5047 scope.go:122] "RemoveContainer" containerID="593f43a62fb8913eb5a0146559b68fce4aa948e2892e537b566a58156dfe9cf9" Oct 30 20:01:35.043431 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:01:35.043397 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"593f43a62fb8913eb5a0146559b68fce4aa948e2892e537b566a58156dfe9cf9\": container with ID starting with 593f43a62fb8913eb5a0146559b68fce4aa948e2892e537b566a58156dfe9cf9 not found: ID does not exist" containerID="593f43a62fb8913eb5a0146559b68fce4aa948e2892e537b566a58156dfe9cf9" Oct 30 20:01:35.043557 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:35.043445 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"593f43a62fb8913eb5a0146559b68fce4aa948e2892e537b566a58156dfe9cf9"} err="failed to get container status \"593f43a62fb8913eb5a0146559b68fce4aa948e2892e537b566a58156dfe9cf9\": rpc error: code = NotFound desc = could not find container \"593f43a62fb8913eb5a0146559b68fce4aa948e2892e537b566a58156dfe9cf9\": container with ID starting with 593f43a62fb8913eb5a0146559b68fce4aa948e2892e537b566a58156dfe9cf9 not found: ID does not exist" Oct 30 20:01:35.081376 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:35.081336 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6"] Oct 30 20:01:35.093045 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:35.092994 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5ac614-5d94969f59-klzn6"] Oct 30 20:01:37.051181 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:37.051146 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45e50f1-ccb5-4cdc-bd7b-75d85074b002" path="/var/lib/kubelet/pods/e45e50f1-ccb5-4cdc-bd7b-75d85074b002/volumes" Oct 30 20:01:38.484245 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.484184 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g"] Oct 30 20:01:38.487080 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487044 5047 cpu_manager.go:414] "containerMap: removing container" podUID="8856e033-793e-4e57-abbf-ae7ae07cbff4" containerName="istio-proxy" Oct 30 20:01:38.487080 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487079 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8856e033-793e-4e57-abbf-ae7ae07cbff4" containerName="istio-proxy" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487100 5047 cpu_manager.go:414] "containerMap: removing container" podUID="6092600c-64f3-401b-a6a6-7c322bfbfb28" containerName="payload-pre-processing" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487108 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6092600c-64f3-401b-a6a6-7c322bfbfb28" containerName="payload-pre-processing" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487131 5047 cpu_manager.go:414] "containerMap: removing container" podUID="1258562d-1c87-43fe-b887-42591c2460d6" containerName="payload-pre-processing" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487138 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1258562d-1c87-43fe-b887-42591c2460d6" containerName="payload-pre-processing" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487148 5047 cpu_manager.go:414] "containerMap: removing container" podUID="c4e3fe64-b5fe-453a-ae2b-7dec205c7931" containerName="istio-proxy" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487156 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e3fe64-b5fe-453a-ae2b-7dec205c7931" containerName="istio-proxy" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487174 5047 cpu_manager.go:414] "containerMap: removing container" podUID="e5345813-fe79-4f6a-a6dc-489770c4e2f6" containerName="istio-proxy" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487180 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5345813-fe79-4f6a-a6dc-489770c4e2f6" containerName="istio-proxy" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487190 5047 cpu_manager.go:414] "containerMap: removing container" podUID="874ca58f-1ef6-4395-bc98-892403898cf0" containerName="maas-api" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487196 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874ca58f-1ef6-4395-bc98-892403898cf0" containerName="maas-api" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487213 5047 cpu_manager.go:414] "containerMap: removing container" podUID="e5345813-fe79-4f6a-a6dc-489770c4e2f6" containerName="istio-proxy" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487220 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5345813-fe79-4f6a-a6dc-489770c4e2f6" containerName="istio-proxy" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487245 5047 cpu_manager.go:414] "containerMap: removing container" podUID="54643a49-f1da-4957-a6d6-81fb67991b2e" containerName="payload-processing" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487252 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54643a49-f1da-4957-a6d6-81fb67991b2e" containerName="payload-processing" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487266 5047 cpu_manager.go:414] "containerMap: removing container" podUID="f10c60fd-c6ab-4d3e-8808-15a141781b2c" containerName="cleanup" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487272 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10c60fd-c6ab-4d3e-8808-15a141781b2c" containerName="cleanup" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487285 5047 cpu_manager.go:414] "containerMap: removing container" podUID="e45e50f1-ccb5-4cdc-bd7b-75d85074b002" containerName="maas-api" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487293 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45e50f1-ccb5-4cdc-bd7b-75d85074b002" containerName="maas-api" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487306 5047 cpu_manager.go:414] "containerMap: removing container" podUID="996764e6-cacf-4b77-bc37-8c0323ce29b5" containerName="payload-processing" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487313 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996764e6-cacf-4b77-bc37-8c0323ce29b5" containerName="payload-processing" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487328 5047 cpu_manager.go:414] "containerMap: removing container" podUID="c4e3fe64-b5fe-453a-ae2b-7dec205c7931" containerName="istio-proxy" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487336 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e3fe64-b5fe-453a-ae2b-7dec205c7931" containerName="istio-proxy" Oct 30 20:01:38.487332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487345 5047 cpu_manager.go:414] "containerMap: removing container" podUID="01aec079-7f44-4c06-98a5-cb46a0efeb3b" containerName="istio-proxy" Oct 30 20:01:38.488490 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487352 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01aec079-7f44-4c06-98a5-cb46a0efeb3b" containerName="istio-proxy" Oct 30 20:01:38.488490 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487363 5047 cpu_manager.go:414] "containerMap: removing container" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" Oct 30 20:01:38.488490 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487370 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" Oct 30 20:01:38.488490 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487501 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="54643a49-f1da-4957-a6d6-81fb67991b2e" containerName="payload-processing" Oct 30 20:01:38.488490 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487516 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45e50f1-ccb5-4cdc-bd7b-75d85074b002" containerName="maas-api" Oct 30 20:01:38.488490 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487526 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="996764e6-cacf-4b77-bc37-8c0323ce29b5" containerName="payload-processing" Oct 30 20:01:38.488490 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487538 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e3fe64-b5fe-453a-ae2b-7dec205c7931" containerName="istio-proxy" Oct 30 20:01:38.488490 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487547 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="01aec079-7f44-4c06-98a5-cb46a0efeb3b" containerName="istio-proxy" Oct 30 20:01:38.488490 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487557 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="9325c7d7-408b-44dc-a660-9f15ec74f9e6" containerName="main" Oct 30 20:01:38.488490 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487566 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="8856e033-793e-4e57-abbf-ae7ae07cbff4" containerName="istio-proxy" Oct 30 20:01:38.488490 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487576 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="6092600c-64f3-401b-a6a6-7c322bfbfb28" containerName="payload-pre-processing" Oct 30 20:01:38.488490 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487589 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="1258562d-1c87-43fe-b887-42591c2460d6" containerName="payload-pre-processing" Oct 30 20:01:38.488490 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487599 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5345813-fe79-4f6a-a6dc-489770c4e2f6" containerName="istio-proxy" Oct 30 20:01:38.488490 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487609 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="874ca58f-1ef6-4395-bc98-892403898cf0" containerName="maas-api" Oct 30 20:01:38.488490 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.487617 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5345813-fe79-4f6a-a6dc-489770c4e2f6" containerName="istio-proxy" Oct 30 20:01:38.495197 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.495162 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" Oct 30 20:01:38.507658 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.507626 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g"] Oct 30 20:01:38.521761 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.521727 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a76aa2-dockercfg-hwm5f\"" Oct 30 20:01:38.521996 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.521974 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a76aa2\"" Oct 30 20:01:38.561395 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.561360 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845e76e9-51c7-440d-a89e-60adc522d008-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g\" (UID: \"845e76e9-51c7-440d-a89e-60adc522d008\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" Oct 30 20:01:38.561578 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.561424 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmfts\" (UniqueName: \"kubernetes.io/projected/845e76e9-51c7-440d-a89e-60adc522d008-kube-api-access-wmfts\") pod \"payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g\" (UID: \"845e76e9-51c7-440d-a89e-60adc522d008\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" Oct 30 20:01:38.643621 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.643586 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff"] Oct 30 20:01:38.644321 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.644300 5047 cpu_manager.go:414] "containerMap: removing container" podUID="8856e033-793e-4e57-abbf-ae7ae07cbff4" containerName="istio-proxy" Oct 30 20:01:38.644321 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.644318 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8856e033-793e-4e57-abbf-ae7ae07cbff4" containerName="istio-proxy" Oct 30 20:01:38.644505 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.644434 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e3fe64-b5fe-453a-ae2b-7dec205c7931" containerName="istio-proxy" Oct 30 20:01:38.644505 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.644448 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="8856e033-793e-4e57-abbf-ae7ae07cbff4" containerName="istio-proxy" Oct 30 20:01:38.648072 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.648053 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff" Oct 30 20:01:38.662481 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.662451 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845e76e9-51c7-440d-a89e-60adc522d008-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g\" (UID: \"845e76e9-51c7-440d-a89e-60adc522d008\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" Oct 30 20:01:38.662615 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.662498 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac686d8-338e-43b4-81b0-15718ed61608-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff\" (UID: \"2ac686d8-338e-43b4-81b0-15718ed61608\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff" Oct 30 20:01:38.662615 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.662546 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmfts\" (UniqueName: \"kubernetes.io/projected/845e76e9-51c7-440d-a89e-60adc522d008-kube-api-access-wmfts\") pod \"payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g\" (UID: \"845e76e9-51c7-440d-a89e-60adc522d008\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" Oct 30 20:01:38.662747 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.662705 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx6bc\" (UniqueName: \"kubernetes.io/projected/2ac686d8-338e-43b4-81b0-15718ed61608-kube-api-access-xx6bc\") pod \"payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff\" (UID: \"2ac686d8-338e-43b4-81b0-15718ed61608\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff" Oct 30 20:01:38.663347 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.663321 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845e76e9-51c7-440d-a89e-60adc522d008-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g\" (UID: \"845e76e9-51c7-440d-a89e-60adc522d008\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" Oct 30 20:01:38.665945 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.665925 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff"] Oct 30 20:01:38.680861 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.680829 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmfts\" (UniqueName: \"kubernetes.io/projected/845e76e9-51c7-440d-a89e-60adc522d008-kube-api-access-wmfts\") pod \"payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g\" (UID: \"845e76e9-51c7-440d-a89e-60adc522d008\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" Oct 30 20:01:38.764125 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.763994 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac686d8-338e-43b4-81b0-15718ed61608-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff\" (UID: \"2ac686d8-338e-43b4-81b0-15718ed61608\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff" Oct 30 20:01:38.764313 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.764170 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx6bc\" (UniqueName: \"kubernetes.io/projected/2ac686d8-338e-43b4-81b0-15718ed61608-kube-api-access-xx6bc\") pod \"payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff\" (UID: \"2ac686d8-338e-43b4-81b0-15718ed61608\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff" Oct 30 20:01:38.764823 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.764794 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac686d8-338e-43b4-81b0-15718ed61608-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff\" (UID: \"2ac686d8-338e-43b4-81b0-15718ed61608\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff" Oct 30 20:01:38.781801 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.781770 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx6bc\" (UniqueName: \"kubernetes.io/projected/2ac686d8-338e-43b4-81b0-15718ed61608-kube-api-access-xx6bc\") pod \"payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff\" (UID: \"2ac686d8-338e-43b4-81b0-15718ed61608\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff" Oct 30 20:01:38.826493 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.826455 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" Oct 30 20:01:38.962672 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:38.962633 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff" Oct 30 20:01:39.128272 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:39.128239 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g"] Oct 30 20:01:39.129711 ip-10-0-140-235 kubenswrapper[5047]: W1030 20:01:39.129681 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845e76e9_51c7_440d_a89e_60adc522d008.slice/crio-8805e0d872a78035b3336c90f30d74d6cbce4e32c1f3a6a0d75840c92f9976f6 WatchSource:0}: Error finding container 8805e0d872a78035b3336c90f30d74d6cbce4e32c1f3a6a0d75840c92f9976f6: Status 404 returned error can't find the container with id 8805e0d872a78035b3336c90f30d74d6cbce4e32c1f3a6a0d75840c92f9976f6 Oct 30 20:01:39.292230 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:39.292198 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff"] Oct 30 20:01:39.293454 ip-10-0-140-235 kubenswrapper[5047]: W1030 20:01:39.293421 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac686d8_338e_43b4_81b0_15718ed61608.slice/crio-2eb451df9a8e4050500ece372926ee02b7e966bf2532ca586cbc9116445490b8 WatchSource:0}: Error finding container 2eb451df9a8e4050500ece372926ee02b7e966bf2532ca586cbc9116445490b8: Status 404 returned error can't find the container with id 2eb451df9a8e4050500ece372926ee02b7e966bf2532ca586cbc9116445490b8 Oct 30 20:01:39.604243 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:01:39.604207 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:01:39.990128 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:39.990035 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj"] Oct 30 20:01:39.990598 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:39.990566 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" podUID="c0cd76c6-9b67-4719-9d13-82315cd9f77a" containerName="istio-proxy" containerID="cri-o://869409a721cfd2f3b4a960818549deec0a34d3edcbcb9adddc1abe3e761fd5c1" gracePeriod=30 Oct 30 20:01:40.133291 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:40.133253 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" event={"ID":"845e76e9-51c7-440d-a89e-60adc522d008","Type":"ContainerStarted","Data":"b21cdb35f6eb3f1d36fc5eb18699fef34297a35d5692aba97ed9043bb418847a"} Oct 30 20:01:40.133594 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:40.133319 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" event={"ID":"845e76e9-51c7-440d-a89e-60adc522d008","Type":"ContainerStarted","Data":"8805e0d872a78035b3336c90f30d74d6cbce4e32c1f3a6a0d75840c92f9976f6"} Oct 30 20:01:40.134312 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:40.134287 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" Oct 30 20:01:40.150513 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:40.149420 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff" event={"ID":"2ac686d8-338e-43b4-81b0-15718ed61608","Type":"ContainerStarted","Data":"37fff8009a6251d4f9224ee1423aec63b3be3228dff3b9255f4ecb26b3aa5e7a"} Oct 30 20:01:40.150513 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:40.149482 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff" event={"ID":"2ac686d8-338e-43b4-81b0-15718ed61608","Type":"ContainerStarted","Data":"2eb451df9a8e4050500ece372926ee02b7e966bf2532ca586cbc9116445490b8"} Oct 30 20:01:40.150513 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:40.150405 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff" Oct 30 20:01:40.180727 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:40.180673 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" podStartSLOduration=2.180659163 podStartE2EDuration="2.180659163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:40.176450474 +0000 UTC m=+3453.250905834" watchObservedRunningTime="2026-10-30 20:01:40.180659163 +0000 UTC m=+3453.255114521" Oct 30 20:01:40.213307 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:40.213244 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff" podStartSLOduration=2.213227664 podStartE2EDuration="2.213227664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:40.212679788 +0000 UTC m=+3453.287135148" watchObservedRunningTime="2026-10-30 20:01:40.213227664 +0000 UTC m=+3453.287683023" Oct 30 20:01:44.721430 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:44.721390 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx"] Oct 30 20:01:44.721862 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:44.721624 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" podUID="e1f14230-b23c-42c4-9709-f67299e49c68" containerName="istio-proxy" containerID="cri-o://afb8373164fc3546e76595177e3ab318047bc52a0103bae326a7a2aa0b077850" gracePeriod=30 Oct 30 20:01:45.223296 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.223263 5047 generic.go:356] "Generic (PLEG): container finished" podID="c0cd76c6-9b67-4719-9d13-82315cd9f77a" containerID="869409a721cfd2f3b4a960818549deec0a34d3edcbcb9adddc1abe3e761fd5c1" exitCode=0 Oct 30 20:01:45.223538 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.223333 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" event={"ID":"c0cd76c6-9b67-4719-9d13-82315cd9f77a","Type":"ContainerDied","Data":"869409a721cfd2f3b4a960818549deec0a34d3edcbcb9adddc1abe3e761fd5c1"} Oct 30 20:01:45.273883 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.273824 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 20:01:45.370204 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.370172 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-podinfo\") pod \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " Oct 30 20:01:45.370377 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.370239 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-workload-certs\") pod \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " Oct 30 20:01:45.370377 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.370271 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-envoy\") pod \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " Oct 30 20:01:45.370377 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.370327 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istiod-ca-cert\") pod \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " Oct 30 20:01:45.370520 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.370389 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-data\") pod \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " Oct 30 20:01:45.370520 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.370446 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-credential-socket\") pod \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " Oct 30 20:01:45.370638 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.370553 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0cd76c6-9b67-4719-9d13-82315cd9f77a-kube-api-access-j5hbk\" (UniqueName: \"kubernetes.io/projected/c0cd76c6-9b67-4719-9d13-82315cd9f77a-kube-api-access-j5hbk\") pod \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " Oct 30 20:01:45.370638 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.370602 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-workload-certs" pod "c0cd76c6-9b67-4719-9d13-82315cd9f77a" (UID: "c0cd76c6-9b67-4719-9d13-82315cd9f77a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:45.370750 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.370613 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-workload-socket\") pod \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " Oct 30 20:01:45.370750 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.370701 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-token\" (UniqueName: \"kubernetes.io/projected/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-token\") pod \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\" (UID: \"c0cd76c6-9b67-4719-9d13-82315cd9f77a\") " Oct 30 20:01:45.370861 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.370744 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-data" pod "c0cd76c6-9b67-4719-9d13-82315cd9f77a" (UID: "c0cd76c6-9b67-4719-9d13-82315cd9f77a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:45.370861 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.370808 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-workload-socket" pod "c0cd76c6-9b67-4719-9d13-82315cd9f77a" (UID: "c0cd76c6-9b67-4719-9d13-82315cd9f77a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:45.371269 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.371244 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:45.371269 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.371271 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:45.371467 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.371268 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-credential-socket" pod "c0cd76c6-9b67-4719-9d13-82315cd9f77a" (UID: "c0cd76c6-9b67-4719-9d13-82315cd9f77a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:45.371467 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.371284 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:45.371467 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.371442 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istiod-ca-cert" pod "c0cd76c6-9b67-4719-9d13-82315cd9f77a" (UID: "c0cd76c6-9b67-4719-9d13-82315cd9f77a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:45.373095 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.373064 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-token" pod "c0cd76c6-9b67-4719-9d13-82315cd9f77a" (UID: "c0cd76c6-9b67-4719-9d13-82315cd9f77a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:45.373095 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.373081 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-envoy" pod "c0cd76c6-9b67-4719-9d13-82315cd9f77a" (UID: "c0cd76c6-9b67-4719-9d13-82315cd9f77a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:45.373277 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.373134 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cd76c6-9b67-4719-9d13-82315cd9f77a-kube-api-access-j5hbk" pod "c0cd76c6-9b67-4719-9d13-82315cd9f77a" (UID: "c0cd76c6-9b67-4719-9d13-82315cd9f77a"). InnerVolumeSpecName "kube-api-access-j5hbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:45.373277 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.373207 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-podinfo" pod "c0cd76c6-9b67-4719-9d13-82315cd9f77a" (UID: "c0cd76c6-9b67-4719-9d13-82315cd9f77a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:01:45.472760 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.472724 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:45.472760 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.472755 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:45.472760 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.472764 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:45.472991 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.472773 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0cd76c6-9b67-4719-9d13-82315cd9f77a-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:45.472991 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.472783 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5hbk\" (UniqueName: \"kubernetes.io/projected/c0cd76c6-9b67-4719-9d13-82315cd9f77a-kube-api-access-j5hbk\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:45.472991 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.472791 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0cd76c6-9b67-4719-9d13-82315cd9f77a-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:45.946342 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.946309 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff"] Oct 30 20:01:45.946743 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:45.946533 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" podUID="0c851817-5127-4d01-91db-5f6bf9e8a062" containerName="payload-processing" containerID="cri-o://da31b0f320d251bf4d850a8d683602d67c122471a95cea3e4c685b499381a20a" gracePeriod=30 Oct 30 20:01:46.232581 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:46.232540 5047 generic.go:356] "Generic (PLEG): container finished" podID="0c851817-5127-4d01-91db-5f6bf9e8a062" containerID="da31b0f320d251bf4d850a8d683602d67c122471a95cea3e4c685b499381a20a" exitCode=0 Oct 30 20:01:46.232770 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:46.232738 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" event={"ID":"0c851817-5127-4d01-91db-5f6bf9e8a062","Type":"ContainerDied","Data":"da31b0f320d251bf4d850a8d683602d67c122471a95cea3e4c685b499381a20a"} Oct 30 20:01:46.232940 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:46.232813 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" event={"ID":"0c851817-5127-4d01-91db-5f6bf9e8a062","Type":"ContainerDied","Data":"12333c69576d536e43723d79ce7cf7ae0e13b3af497a7da1f157cc9c19cf3c5a"} Oct 30 20:01:46.232940 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:46.232843 5047 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12333c69576d536e43723d79ce7cf7ae0e13b3af497a7da1f157cc9c19cf3c5a" Oct 30 20:01:46.234666 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:46.234644 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" event={"ID":"c0cd76c6-9b67-4719-9d13-82315cd9f77a","Type":"ContainerDied","Data":"fcd3d12656d612173de2fec06f288fb140f304511b44eb828247b9805e1c92ed"} Oct 30 20:01:46.234754 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:46.234697 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj" Oct 30 20:01:46.234807 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:46.234698 5047 scope.go:122] "RemoveContainer" containerID="869409a721cfd2f3b4a960818549deec0a34d3edcbcb9adddc1abe3e761fd5c1" Oct 30 20:01:46.309055 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:46.308132 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" Oct 30 20:01:46.345315 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:46.345083 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj"] Oct 30 20:01:46.354828 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:46.354791 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cd4bde-openshift-default-5b8b8d586c-mtmdj"] Oct 30 20:01:46.391486 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:46.391443 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c851817-5127-4d01-91db-5f6bf9e8a062-kube-api-access-mvjzb\" (UniqueName: \"kubernetes.io/projected/0c851817-5127-4d01-91db-5f6bf9e8a062-kube-api-access-mvjzb\") pod \"0c851817-5127-4d01-91db-5f6bf9e8a062\" (UID: \"0c851817-5127-4d01-91db-5f6bf9e8a062\") " Oct 30 20:01:46.391684 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:46.391499 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c851817-5127-4d01-91db-5f6bf9e8a062-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c851817-5127-4d01-91db-5f6bf9e8a062-plugins-config-volume\") pod \"0c851817-5127-4d01-91db-5f6bf9e8a062\" (UID: \"0c851817-5127-4d01-91db-5f6bf9e8a062\") " Oct 30 20:01:46.391991 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:46.391966 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c851817-5127-4d01-91db-5f6bf9e8a062-plugins-config-volume" pod "0c851817-5127-4d01-91db-5f6bf9e8a062" (UID: "0c851817-5127-4d01-91db-5f6bf9e8a062"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:46.393940 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:46.393915 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c851817-5127-4d01-91db-5f6bf9e8a062-kube-api-access-mvjzb" pod "0c851817-5127-4d01-91db-5f6bf9e8a062" (UID: "0c851817-5127-4d01-91db-5f6bf9e8a062"). InnerVolumeSpecName "kube-api-access-mvjzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:46.493590 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:46.493516 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvjzb\" (UniqueName: \"kubernetes.io/projected/0c851817-5127-4d01-91db-5f6bf9e8a062-kube-api-access-mvjzb\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:46.493590 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:46.493547 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c851817-5127-4d01-91db-5f6bf9e8a062-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:47.051950 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:47.051913 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0cd76c6-9b67-4719-9d13-82315cd9f77a" path="/var/lib/kubelet/pods/c0cd76c6-9b67-4719-9d13-82315cd9f77a/volumes" Oct 30 20:01:47.241904 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:47.241873 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff" Oct 30 20:01:47.311768 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:47.311688 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff"] Oct 30 20:01:47.321709 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:47.321679 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2afc11-7fdb7cbfcb-2ktff"] Oct 30 20:01:49.050429 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:49.050345 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c851817-5127-4d01-91db-5f6bf9e8a062" path="/var/lib/kubelet/pods/0c851817-5127-4d01-91db-5f6bf9e8a062/volumes" Oct 30 20:01:50.009226 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.009202 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 20:01:50.139096 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.139066 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f14230-b23c-42c4-9709-f67299e49c68-kube-api-access-d4dh5\" (UniqueName: \"kubernetes.io/projected/e1f14230-b23c-42c4-9709-f67299e49c68-kube-api-access-d4dh5\") pod \"e1f14230-b23c-42c4-9709-f67299e49c68\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " Oct 30 20:01:50.139544 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.139110 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1f14230-b23c-42c4-9709-f67299e49c68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f14230-b23c-42c4-9709-f67299e49c68-istiod-ca-cert\") pod \"e1f14230-b23c-42c4-9709-f67299e49c68\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " Oct 30 20:01:50.139544 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.139192 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-credential-socket\") pod \"e1f14230-b23c-42c4-9709-f67299e49c68\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " Oct 30 20:01:50.139544 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.139215 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-workload-certs\") pod \"e1f14230-b23c-42c4-9709-f67299e49c68\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " Oct 30 20:01:50.139544 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.139259 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1f14230-b23c-42c4-9709-f67299e49c68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f14230-b23c-42c4-9709-f67299e49c68-istio-podinfo\") pod \"e1f14230-b23c-42c4-9709-f67299e49c68\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " Oct 30 20:01:50.139544 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.139276 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-workload-socket\") pod \"e1f14230-b23c-42c4-9709-f67299e49c68\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " Oct 30 20:01:50.139544 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.139324 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f14230-b23c-42c4-9709-f67299e49c68-istio-token\" (UniqueName: \"kubernetes.io/projected/e1f14230-b23c-42c4-9709-f67299e49c68-istio-token\") pod \"e1f14230-b23c-42c4-9709-f67299e49c68\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " Oct 30 20:01:50.139544 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.139394 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-istio-data\") pod \"e1f14230-b23c-42c4-9709-f67299e49c68\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " Oct 30 20:01:50.139544 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.139432 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-istio-envoy\") pod \"e1f14230-b23c-42c4-9709-f67299e49c68\" (UID: \"e1f14230-b23c-42c4-9709-f67299e49c68\") " Oct 30 20:01:50.140035 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.139575 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-workload-certs" pod "e1f14230-b23c-42c4-9709-f67299e49c68" (UID: "e1f14230-b23c-42c4-9709-f67299e49c68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:50.140035 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.139588 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f14230-b23c-42c4-9709-f67299e49c68-istiod-ca-cert" pod "e1f14230-b23c-42c4-9709-f67299e49c68" (UID: "e1f14230-b23c-42c4-9709-f67299e49c68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:50.140035 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.139837 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-istio-data" pod "e1f14230-b23c-42c4-9709-f67299e49c68" (UID: "e1f14230-b23c-42c4-9709-f67299e49c68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:50.140035 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.139918 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-workload-socket" pod "e1f14230-b23c-42c4-9709-f67299e49c68" (UID: "e1f14230-b23c-42c4-9709-f67299e49c68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:50.140274 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.140038 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-credential-socket" pod "e1f14230-b23c-42c4-9709-f67299e49c68" (UID: "e1f14230-b23c-42c4-9709-f67299e49c68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:50.140274 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.140219 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f14230-b23c-42c4-9709-f67299e49c68-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:50.140274 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.140239 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:50.140274 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.140255 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:50.140274 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.140268 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:50.140565 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.140282 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:50.141571 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.141543 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f14230-b23c-42c4-9709-f67299e49c68-kube-api-access-d4dh5" pod "e1f14230-b23c-42c4-9709-f67299e49c68" (UID: "e1f14230-b23c-42c4-9709-f67299e49c68"). InnerVolumeSpecName "kube-api-access-d4dh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:50.141692 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.141659 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1f14230-b23c-42c4-9709-f67299e49c68-istio-podinfo" pod "e1f14230-b23c-42c4-9709-f67299e49c68" (UID: "e1f14230-b23c-42c4-9709-f67299e49c68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:01:50.141874 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.141853 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-istio-envoy" pod "e1f14230-b23c-42c4-9709-f67299e49c68" (UID: "e1f14230-b23c-42c4-9709-f67299e49c68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:50.142001 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.141983 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f14230-b23c-42c4-9709-f67299e49c68-istio-token" pod "e1f14230-b23c-42c4-9709-f67299e49c68" (UID: "e1f14230-b23c-42c4-9709-f67299e49c68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:50.241050 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.240961 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4dh5\" (UniqueName: \"kubernetes.io/projected/e1f14230-b23c-42c4-9709-f67299e49c68-kube-api-access-d4dh5\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:50.241050 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.240989 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f14230-b23c-42c4-9709-f67299e49c68-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:50.241050 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.240998 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f14230-b23c-42c4-9709-f67299e49c68-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:50.241050 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.241041 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f14230-b23c-42c4-9709-f67299e49c68-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:01:50.261955 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.261925 5047 generic.go:356] "Generic (PLEG): container finished" podID="e1f14230-b23c-42c4-9709-f67299e49c68" containerID="afb8373164fc3546e76595177e3ab318047bc52a0103bae326a7a2aa0b077850" exitCode=0 Oct 30 20:01:50.262150 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.261976 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" Oct 30 20:01:50.262150 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.262046 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" event={"ID":"e1f14230-b23c-42c4-9709-f67299e49c68","Type":"ContainerDied","Data":"afb8373164fc3546e76595177e3ab318047bc52a0103bae326a7a2aa0b077850"} Oct 30 20:01:50.262150 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.262099 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" event={"ID":"e1f14230-b23c-42c4-9709-f67299e49c68","Type":"ContainerDied","Data":"cc4a8aec2ee3f2e4d689ce936c9ac2f91005fe262405009541865ad8745c9207"} Oct 30 20:01:50.262150 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.262130 5047 scope.go:122] "RemoveContainer" containerID="afb8373164fc3546e76595177e3ab318047bc52a0103bae326a7a2aa0b077850" Oct 30 20:01:50.273755 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.273732 5047 scope.go:122] "RemoveContainer" containerID="afb8373164fc3546e76595177e3ab318047bc52a0103bae326a7a2aa0b077850" Oct 30 20:01:50.274140 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:01:50.274119 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb8373164fc3546e76595177e3ab318047bc52a0103bae326a7a2aa0b077850\": container with ID starting with afb8373164fc3546e76595177e3ab318047bc52a0103bae326a7a2aa0b077850 not found: ID does not exist" containerID="afb8373164fc3546e76595177e3ab318047bc52a0103bae326a7a2aa0b077850" Oct 30 20:01:50.274226 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.274153 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb8373164fc3546e76595177e3ab318047bc52a0103bae326a7a2aa0b077850"} err="failed to get container status \"afb8373164fc3546e76595177e3ab318047bc52a0103bae326a7a2aa0b077850\": rpc error: code = NotFound desc = could not find container \"afb8373164fc3546e76595177e3ab318047bc52a0103bae326a7a2aa0b077850\": container with ID starting with afb8373164fc3546e76595177e3ab318047bc52a0103bae326a7a2aa0b077850 not found: ID does not exist" Oct 30 20:01:50.308424 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.308384 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx"] Oct 30 20:01:50.311150 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.311130 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx"] Oct 30 20:01:50.998709 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.998674 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:01:50.998919 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:50.998748 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-rk8fx" podUID="e1f14230-b23c-42c4-9709-f67299e49c68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:51.050789 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:51.050753 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1f14230-b23c-42c4-9709-f67299e49c68" path="/var/lib/kubelet/pods/e1f14230-b23c-42c4-9709-f67299e49c68/volumes" Oct 30 20:01:52.222230 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:52.222194 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" Oct 30 20:01:52.223156 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:01:52.222286 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff" Oct 30 20:02:02.293096 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:02.293057 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf"] Oct 30 20:02:02.293591 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:02.293279 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" podUID="2123dbbb-db97-46f3-b265-a81414f0652c" containerName="istio-proxy" containerID="cri-o://a317c07c3b6d43c6dd7b7f5eb3306529cf080af30733095e8f40c55326b35b28" gracePeriod=30 Oct 30 20:02:03.212027 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.211974 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv"] Oct 30 20:02:03.212587 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.212531 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" podUID="5a2dff4c-72fb-4569-a49a-f8995370ad75" containerName="maas-api" containerID="cri-o://09ecea5b6923104c1000d777944b42649f6d5bd5afbd8247aa272a4d712716bc" gracePeriod=30 Oct 30 20:02:03.359435 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.359399 5047 generic.go:356] "Generic (PLEG): container finished" podID="5a2dff4c-72fb-4569-a49a-f8995370ad75" containerID="09ecea5b6923104c1000d777944b42649f6d5bd5afbd8247aa272a4d712716bc" exitCode=0 Oct 30 20:02:03.359894 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.359633 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" event={"ID":"5a2dff4c-72fb-4569-a49a-f8995370ad75","Type":"ContainerDied","Data":"09ecea5b6923104c1000d777944b42649f6d5bd5afbd8247aa272a4d712716bc"} Oct 30 20:02:03.362334 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.362296 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l"] Oct 30 20:02:03.362591 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.362522 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" podUID="3e35030c-45c1-4c40-98e8-8a4a1fe5b72f" containerName="payload-processing" containerID="cri-o://313adebc5edec1e24855657b353a42edf64a74bb5cdd7ac5e8d052c395262c18" gracePeriod=30 Oct 30 20:02:03.380381 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.380346 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q"] Oct 30 20:02:03.380599 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.380559 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" podUID="d764690a-8072-41fb-b021-1c6ad226146f" containerName="payload-pre-processing" containerID="cri-o://9ddd2a1cdcdf9190bea59a337f5a98b958b185e47c4c48d8c6c1a7d58e9aeac9" gracePeriod=30 Oct 30 20:02:03.569528 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.569495 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 20:02:03.667963 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.666576 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a2dff4c-72fb-4569-a49a-f8995370ad75-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a2dff4c-72fb-4569-a49a-f8995370ad75-openshift-service-ca\") pod \"5a2dff4c-72fb-4569-a49a-f8995370ad75\" (UID: \"5a2dff4c-72fb-4569-a49a-f8995370ad75\") " Oct 30 20:02:03.667963 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.666681 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a2dff4c-72fb-4569-a49a-f8995370ad75-kube-api-access-v6nw2\" (UniqueName: \"kubernetes.io/projected/5a2dff4c-72fb-4569-a49a-f8995370ad75-kube-api-access-v6nw2\") pod \"5a2dff4c-72fb-4569-a49a-f8995370ad75\" (UID: \"5a2dff4c-72fb-4569-a49a-f8995370ad75\") " Oct 30 20:02:03.667963 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.666774 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a2dff4c-72fb-4569-a49a-f8995370ad75-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a2dff4c-72fb-4569-a49a-f8995370ad75-metrics-certs\") pod \"5a2dff4c-72fb-4569-a49a-f8995370ad75\" (UID: \"5a2dff4c-72fb-4569-a49a-f8995370ad75\") " Oct 30 20:02:03.667963 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.666849 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a2dff4c-72fb-4569-a49a-f8995370ad75-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a2dff4c-72fb-4569-a49a-f8995370ad75-maas-api-tls\") pod \"5a2dff4c-72fb-4569-a49a-f8995370ad75\" (UID: \"5a2dff4c-72fb-4569-a49a-f8995370ad75\") " Oct 30 20:02:03.670374 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.670341 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a2dff4c-72fb-4569-a49a-f8995370ad75-openshift-service-ca" pod "5a2dff4c-72fb-4569-a49a-f8995370ad75" (UID: "5a2dff4c-72fb-4569-a49a-f8995370ad75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:03.674426 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.674399 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a2dff4c-72fb-4569-a49a-f8995370ad75-metrics-certs" pod "5a2dff4c-72fb-4569-a49a-f8995370ad75" (UID: "5a2dff4c-72fb-4569-a49a-f8995370ad75"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:03.677478 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.677439 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a2dff4c-72fb-4569-a49a-f8995370ad75-maas-api-tls" pod "5a2dff4c-72fb-4569-a49a-f8995370ad75" (UID: "5a2dff4c-72fb-4569-a49a-f8995370ad75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:03.679372 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.679346 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2dff4c-72fb-4569-a49a-f8995370ad75-kube-api-access-v6nw2" pod "5a2dff4c-72fb-4569-a49a-f8995370ad75" (UID: "5a2dff4c-72fb-4569-a49a-f8995370ad75"). InnerVolumeSpecName "kube-api-access-v6nw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:03.698864 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.698831 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" Oct 30 20:02:03.777087 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.776945 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a2dff4c-72fb-4569-a49a-f8995370ad75-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:03.778488 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.778141 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6nw2\" (UniqueName: \"kubernetes.io/projected/5a2dff4c-72fb-4569-a49a-f8995370ad75-kube-api-access-v6nw2\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:03.778488 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.778174 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a2dff4c-72fb-4569-a49a-f8995370ad75-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:03.778488 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.778190 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a2dff4c-72fb-4569-a49a-f8995370ad75-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:03.797071 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.796993 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" Oct 30 20:02:03.879704 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.879666 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d764690a-8072-41fb-b021-1c6ad226146f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d764690a-8072-41fb-b021-1c6ad226146f-plugins-config-volume\") pod \"d764690a-8072-41fb-b021-1c6ad226146f\" (UID: \"d764690a-8072-41fb-b021-1c6ad226146f\") " Oct 30 20:02:03.879987 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.879759 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d764690a-8072-41fb-b021-1c6ad226146f-kube-api-access-9mndb\" (UniqueName: \"kubernetes.io/projected/d764690a-8072-41fb-b021-1c6ad226146f-kube-api-access-9mndb\") pod \"d764690a-8072-41fb-b021-1c6ad226146f\" (UID: \"d764690a-8072-41fb-b021-1c6ad226146f\") " Oct 30 20:02:03.879987 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.879810 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e35030c-45c1-4c40-98e8-8a4a1fe5b72f-kube-api-access-jw6n4\" (UniqueName: \"kubernetes.io/projected/3e35030c-45c1-4c40-98e8-8a4a1fe5b72f-kube-api-access-jw6n4\") pod \"3e35030c-45c1-4c40-98e8-8a4a1fe5b72f\" (UID: \"3e35030c-45c1-4c40-98e8-8a4a1fe5b72f\") " Oct 30 20:02:03.879987 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.879848 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e35030c-45c1-4c40-98e8-8a4a1fe5b72f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e35030c-45c1-4c40-98e8-8a4a1fe5b72f-plugins-config-volume\") pod \"3e35030c-45c1-4c40-98e8-8a4a1fe5b72f\" (UID: \"3e35030c-45c1-4c40-98e8-8a4a1fe5b72f\") " Oct 30 20:02:03.880290 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.880128 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d764690a-8072-41fb-b021-1c6ad226146f-plugins-config-volume" pod "d764690a-8072-41fb-b021-1c6ad226146f" (UID: "d764690a-8072-41fb-b021-1c6ad226146f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:03.880503 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.880478 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e35030c-45c1-4c40-98e8-8a4a1fe5b72f-plugins-config-volume" pod "3e35030c-45c1-4c40-98e8-8a4a1fe5b72f" (UID: "3e35030c-45c1-4c40-98e8-8a4a1fe5b72f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:03.880624 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.880602 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d764690a-8072-41fb-b021-1c6ad226146f-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:03.883262 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.883233 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e35030c-45c1-4c40-98e8-8a4a1fe5b72f-kube-api-access-jw6n4" pod "3e35030c-45c1-4c40-98e8-8a4a1fe5b72f" (UID: "3e35030c-45c1-4c40-98e8-8a4a1fe5b72f"). InnerVolumeSpecName "kube-api-access-jw6n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:03.884028 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.883977 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d764690a-8072-41fb-b021-1c6ad226146f-kube-api-access-9mndb" pod "d764690a-8072-41fb-b021-1c6ad226146f" (UID: "d764690a-8072-41fb-b021-1c6ad226146f"). InnerVolumeSpecName "kube-api-access-9mndb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:03.981328 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.981290 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mndb\" (UniqueName: \"kubernetes.io/projected/d764690a-8072-41fb-b021-1c6ad226146f-kube-api-access-9mndb\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:03.981328 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.981332 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw6n4\" (UniqueName: \"kubernetes.io/projected/3e35030c-45c1-4c40-98e8-8a4a1fe5b72f-kube-api-access-jw6n4\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:03.981593 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:03.981343 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e35030c-45c1-4c40-98e8-8a4a1fe5b72f-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:04.368553 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.368506 5047 generic.go:356] "Generic (PLEG): container finished" podID="3e35030c-45c1-4c40-98e8-8a4a1fe5b72f" containerID="313adebc5edec1e24855657b353a42edf64a74bb5cdd7ac5e8d052c395262c18" exitCode=0 Oct 30 20:02:04.368553 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.368557 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" Oct 30 20:02:04.369121 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.368590 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" event={"ID":"3e35030c-45c1-4c40-98e8-8a4a1fe5b72f","Type":"ContainerDied","Data":"313adebc5edec1e24855657b353a42edf64a74bb5cdd7ac5e8d052c395262c18"} Oct 30 20:02:04.369121 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.368659 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l" event={"ID":"3e35030c-45c1-4c40-98e8-8a4a1fe5b72f","Type":"ContainerDied","Data":"020af34bcb336ff1d4d6e36c592992d76bfb975def50527f90a8900ead6b7683"} Oct 30 20:02:04.369121 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.368694 5047 scope.go:122] "RemoveContainer" containerID="313adebc5edec1e24855657b353a42edf64a74bb5cdd7ac5e8d052c395262c18" Oct 30 20:02:04.370108 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.370086 5047 generic.go:356] "Generic (PLEG): container finished" podID="d764690a-8072-41fb-b021-1c6ad226146f" containerID="9ddd2a1cdcdf9190bea59a337f5a98b958b185e47c4c48d8c6c1a7d58e9aeac9" exitCode=0 Oct 30 20:02:04.370191 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.370124 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" Oct 30 20:02:04.370191 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.370162 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" event={"ID":"d764690a-8072-41fb-b021-1c6ad226146f","Type":"ContainerDied","Data":"9ddd2a1cdcdf9190bea59a337f5a98b958b185e47c4c48d8c6c1a7d58e9aeac9"} Oct 30 20:02:04.370285 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.370207 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q" event={"ID":"d764690a-8072-41fb-b021-1c6ad226146f","Type":"ContainerDied","Data":"66d54f9a790ee6556f8a0cad15cd802dd967e698472da69459721322c9fc6645"} Oct 30 20:02:04.371832 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.371806 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" event={"ID":"5a2dff4c-72fb-4569-a49a-f8995370ad75","Type":"ContainerDied","Data":"2bfdd34645ce932c63f84fb8fb6db5e9f3ada10c51c16436c56a37bb15a9deb1"} Oct 30 20:02:04.371978 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.371889 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv" Oct 30 20:02:04.381758 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.381602 5047 scope.go:122] "RemoveContainer" containerID="313adebc5edec1e24855657b353a42edf64a74bb5cdd7ac5e8d052c395262c18" Oct 30 20:02:04.382018 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:02:04.381987 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313adebc5edec1e24855657b353a42edf64a74bb5cdd7ac5e8d052c395262c18\": container with ID starting with 313adebc5edec1e24855657b353a42edf64a74bb5cdd7ac5e8d052c395262c18 not found: ID does not exist" containerID="313adebc5edec1e24855657b353a42edf64a74bb5cdd7ac5e8d052c395262c18" Oct 30 20:02:04.382094 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.382046 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313adebc5edec1e24855657b353a42edf64a74bb5cdd7ac5e8d052c395262c18"} err="failed to get container status \"313adebc5edec1e24855657b353a42edf64a74bb5cdd7ac5e8d052c395262c18\": rpc error: code = NotFound desc = could not find container \"313adebc5edec1e24855657b353a42edf64a74bb5cdd7ac5e8d052c395262c18\": container with ID starting with 313adebc5edec1e24855657b353a42edf64a74bb5cdd7ac5e8d052c395262c18 not found: ID does not exist" Oct 30 20:02:04.382094 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.382063 5047 scope.go:122] "RemoveContainer" containerID="9ddd2a1cdcdf9190bea59a337f5a98b958b185e47c4c48d8c6c1a7d58e9aeac9" Oct 30 20:02:04.391384 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.391367 5047 scope.go:122] "RemoveContainer" containerID="9ddd2a1cdcdf9190bea59a337f5a98b958b185e47c4c48d8c6c1a7d58e9aeac9" Oct 30 20:02:04.391671 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:02:04.391648 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ddd2a1cdcdf9190bea59a337f5a98b958b185e47c4c48d8c6c1a7d58e9aeac9\": container with ID starting with 9ddd2a1cdcdf9190bea59a337f5a98b958b185e47c4c48d8c6c1a7d58e9aeac9 not found: ID does not exist" containerID="9ddd2a1cdcdf9190bea59a337f5a98b958b185e47c4c48d8c6c1a7d58e9aeac9" Oct 30 20:02:04.391735 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.391682 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ddd2a1cdcdf9190bea59a337f5a98b958b185e47c4c48d8c6c1a7d58e9aeac9"} err="failed to get container status \"9ddd2a1cdcdf9190bea59a337f5a98b958b185e47c4c48d8c6c1a7d58e9aeac9\": rpc error: code = NotFound desc = could not find container \"9ddd2a1cdcdf9190bea59a337f5a98b958b185e47c4c48d8c6c1a7d58e9aeac9\": container with ID starting with 9ddd2a1cdcdf9190bea59a337f5a98b958b185e47c4c48d8c6c1a7d58e9aeac9 not found: ID does not exist" Oct 30 20:02:04.391735 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.391704 5047 scope.go:122] "RemoveContainer" containerID="09ecea5b6923104c1000d777944b42649f6d5bd5afbd8247aa272a4d712716bc" Oct 30 20:02:04.438435 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.438406 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv"] Oct 30 20:02:04.447963 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.447942 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-315ea2-78db8dcc44-2rxsv"] Oct 30 20:02:04.472635 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.472604 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q"] Oct 30 20:02:04.477044 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.476995 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-315ea2-5df579ccc9-4nj5q"] Oct 30 20:02:04.505493 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.505437 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l"] Oct 30 20:02:04.510297 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:04.510276 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-315ea2-b47ff788d-h5x2l"] Oct 30 20:02:05.031787 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:05.031754 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk"] Oct 30 20:02:05.049807 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:05.049774 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e35030c-45c1-4c40-98e8-8a4a1fe5b72f" path="/var/lib/kubelet/pods/3e35030c-45c1-4c40-98e8-8a4a1fe5b72f/volumes" Oct 30 20:02:05.050269 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:05.050255 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a2dff4c-72fb-4569-a49a-f8995370ad75" path="/var/lib/kubelet/pods/5a2dff4c-72fb-4569-a49a-f8995370ad75/volumes" Oct 30 20:02:05.050627 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:05.050614 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d764690a-8072-41fb-b021-1c6ad226146f" path="/var/lib/kubelet/pods/d764690a-8072-41fb-b021-1c6ad226146f/volumes" Oct 30 20:02:05.061146 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:05.061123 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45"] Oct 30 20:02:07.578964 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.578933 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 20:02:07.620744 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.620710 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-workload-certs\") pod \"2123dbbb-db97-46f3-b265-a81414f0652c\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " Oct 30 20:02:07.620894 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.620754 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2123dbbb-db97-46f3-b265-a81414f0652c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2123dbbb-db97-46f3-b265-a81414f0652c-istio-podinfo\") pod \"2123dbbb-db97-46f3-b265-a81414f0652c\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " Oct 30 20:02:07.620894 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.620789 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-istio-data\") pod \"2123dbbb-db97-46f3-b265-a81414f0652c\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " Oct 30 20:02:07.620894 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.620808 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-istio-envoy\") pod \"2123dbbb-db97-46f3-b265-a81414f0652c\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " Oct 30 20:02:07.621127 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.620911 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-credential-socket\") pod \"2123dbbb-db97-46f3-b265-a81414f0652c\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " Oct 30 20:02:07.621127 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.620938 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2123dbbb-db97-46f3-b265-a81414f0652c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2123dbbb-db97-46f3-b265-a81414f0652c-istiod-ca-cert\") pod \"2123dbbb-db97-46f3-b265-a81414f0652c\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " Oct 30 20:02:07.621127 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.620974 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-workload-certs" pod "2123dbbb-db97-46f3-b265-a81414f0652c" (UID: "2123dbbb-db97-46f3-b265-a81414f0652c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:07.621127 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.620988 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2123dbbb-db97-46f3-b265-a81414f0652c-kube-api-access-br7h7\" (UniqueName: \"kubernetes.io/projected/2123dbbb-db97-46f3-b265-a81414f0652c-kube-api-access-br7h7\") pod \"2123dbbb-db97-46f3-b265-a81414f0652c\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " Oct 30 20:02:07.621336 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.621196 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2123dbbb-db97-46f3-b265-a81414f0652c-istio-token\" (UniqueName: \"kubernetes.io/projected/2123dbbb-db97-46f3-b265-a81414f0652c-istio-token\") pod \"2123dbbb-db97-46f3-b265-a81414f0652c\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " Oct 30 20:02:07.621336 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.621209 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-istio-data" pod "2123dbbb-db97-46f3-b265-a81414f0652c" (UID: "2123dbbb-db97-46f3-b265-a81414f0652c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:07.621336 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.621246 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-workload-socket\") pod \"2123dbbb-db97-46f3-b265-a81414f0652c\" (UID: \"2123dbbb-db97-46f3-b265-a81414f0652c\") " Oct 30 20:02:07.621336 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.621255 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-credential-socket" pod "2123dbbb-db97-46f3-b265-a81414f0652c" (UID: "2123dbbb-db97-46f3-b265-a81414f0652c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:07.621559 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.621373 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2123dbbb-db97-46f3-b265-a81414f0652c-istiod-ca-cert" pod "2123dbbb-db97-46f3-b265-a81414f0652c" (UID: "2123dbbb-db97-46f3-b265-a81414f0652c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:07.621632 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.621575 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-workload-socket" pod "2123dbbb-db97-46f3-b265-a81414f0652c" (UID: "2123dbbb-db97-46f3-b265-a81414f0652c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:07.621774 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.621758 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:07.621841 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.621779 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2123dbbb-db97-46f3-b265-a81414f0652c-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:07.621841 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.621793 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:07.621841 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.621807 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:07.621841 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.621821 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:07.623346 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.623326 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2123dbbb-db97-46f3-b265-a81414f0652c-istio-token" pod "2123dbbb-db97-46f3-b265-a81414f0652c" (UID: "2123dbbb-db97-46f3-b265-a81414f0652c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:07.623479 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.623448 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2123dbbb-db97-46f3-b265-a81414f0652c-kube-api-access-br7h7" pod "2123dbbb-db97-46f3-b265-a81414f0652c" (UID: "2123dbbb-db97-46f3-b265-a81414f0652c"). InnerVolumeSpecName "kube-api-access-br7h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:07.623584 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.623506 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-istio-envoy" pod "2123dbbb-db97-46f3-b265-a81414f0652c" (UID: "2123dbbb-db97-46f3-b265-a81414f0652c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:07.623584 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.623511 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2123dbbb-db97-46f3-b265-a81414f0652c-istio-podinfo" pod "2123dbbb-db97-46f3-b265-a81414f0652c" (UID: "2123dbbb-db97-46f3-b265-a81414f0652c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:07.722932 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.722895 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2123dbbb-db97-46f3-b265-a81414f0652c-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:07.722932 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.722926 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2123dbbb-db97-46f3-b265-a81414f0652c-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:07.722932 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.722937 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br7h7\" (UniqueName: \"kubernetes.io/projected/2123dbbb-db97-46f3-b265-a81414f0652c-kube-api-access-br7h7\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:07.723204 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:07.722949 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2123dbbb-db97-46f3-b265-a81414f0652c-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:08.405464 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:08.405427 5047 generic.go:356] "Generic (PLEG): container finished" podID="2123dbbb-db97-46f3-b265-a81414f0652c" containerID="a317c07c3b6d43c6dd7b7f5eb3306529cf080af30733095e8f40c55326b35b28" exitCode=0 Oct 30 20:02:08.405657 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:08.405473 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" Oct 30 20:02:08.405657 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:08.405515 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" event={"ID":"2123dbbb-db97-46f3-b265-a81414f0652c","Type":"ContainerDied","Data":"a317c07c3b6d43c6dd7b7f5eb3306529cf080af30733095e8f40c55326b35b28"} Oct 30 20:02:08.405657 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:08.405574 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf" event={"ID":"2123dbbb-db97-46f3-b265-a81414f0652c","Type":"ContainerDied","Data":"dd71ae2669327f8a830b87b36e71f7bca7a88d4323250398f619c0e58e3363ec"} Oct 30 20:02:08.405657 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:08.405604 5047 scope.go:122] "RemoveContainer" containerID="a317c07c3b6d43c6dd7b7f5eb3306529cf080af30733095e8f40c55326b35b28" Oct 30 20:02:08.416783 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:08.416661 5047 scope.go:122] "RemoveContainer" containerID="36890bb1da7aa5b57502bfc2c25147c106279f5ffd1cda27275fd7443e0f8614" Oct 30 20:02:08.427330 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:08.427303 5047 scope.go:122] "RemoveContainer" containerID="a317c07c3b6d43c6dd7b7f5eb3306529cf080af30733095e8f40c55326b35b28" Oct 30 20:02:08.427723 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:02:08.427688 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a317c07c3b6d43c6dd7b7f5eb3306529cf080af30733095e8f40c55326b35b28\": container with ID starting with a317c07c3b6d43c6dd7b7f5eb3306529cf080af30733095e8f40c55326b35b28 not found: ID does not exist" containerID="a317c07c3b6d43c6dd7b7f5eb3306529cf080af30733095e8f40c55326b35b28" Oct 30 20:02:08.427864 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:08.427730 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a317c07c3b6d43c6dd7b7f5eb3306529cf080af30733095e8f40c55326b35b28"} err="failed to get container status \"a317c07c3b6d43c6dd7b7f5eb3306529cf080af30733095e8f40c55326b35b28\": rpc error: code = NotFound desc = could not find container \"a317c07c3b6d43c6dd7b7f5eb3306529cf080af30733095e8f40c55326b35b28\": container with ID starting with a317c07c3b6d43c6dd7b7f5eb3306529cf080af30733095e8f40c55326b35b28 not found: ID does not exist" Oct 30 20:02:08.427864 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:08.427753 5047 scope.go:122] "RemoveContainer" containerID="36890bb1da7aa5b57502bfc2c25147c106279f5ffd1cda27275fd7443e0f8614" Oct 30 20:02:08.428069 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:02:08.428043 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36890bb1da7aa5b57502bfc2c25147c106279f5ffd1cda27275fd7443e0f8614\": container with ID starting with 36890bb1da7aa5b57502bfc2c25147c106279f5ffd1cda27275fd7443e0f8614 not found: ID does not exist" containerID="36890bb1da7aa5b57502bfc2c25147c106279f5ffd1cda27275fd7443e0f8614" Oct 30 20:02:08.428145 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:08.428078 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36890bb1da7aa5b57502bfc2c25147c106279f5ffd1cda27275fd7443e0f8614"} err="failed to get container status \"36890bb1da7aa5b57502bfc2c25147c106279f5ffd1cda27275fd7443e0f8614\": rpc error: code = NotFound desc = could not find container \"36890bb1da7aa5b57502bfc2c25147c106279f5ffd1cda27275fd7443e0f8614\": container with ID starting with 36890bb1da7aa5b57502bfc2c25147c106279f5ffd1cda27275fd7443e0f8614 not found: ID does not exist" Oct 30 20:02:08.462160 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:08.462118 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf"] Oct 30 20:02:08.471053 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:08.471019 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6b7ccc-openshift-default-6db5d9db95-2gsrf"] Oct 30 20:02:09.050688 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:09.050654 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2123dbbb-db97-46f3-b265-a81414f0652c" path="/var/lib/kubelet/pods/2123dbbb-db97-46f3-b265-a81414f0652c/volumes" Oct 30 20:02:09.602340 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:02:09.602302 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:02:10.032551 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:10.032450 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" podUID="87b98550-4b44-4b24-8c55-03a8e05f08df" containerName="payload-processing" containerID="cri-o://86c62f6d2eca6d96ee48d9d091102e0b2eb6fe3e08c3ac42120a954967183d47" gracePeriod=25 Oct 30 20:02:10.061463 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:10.061415 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" podUID="d1e68671-670d-4915-ba4d-47c184994e6a" containerName="payload-pre-processing" containerID="cri-o://96c13429ba605e3212ef6e9f50cd199a7d08a999b13b2b7102abe809e2d0683d" gracePeriod=25 Oct 30 20:02:12.488512 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:12.488471 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" podUID="d1e68671-670d-4915-ba4d-47c184994e6a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:02:12.488964 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:12.488544 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" podUID="87b98550-4b44-4b24-8c55-03a8e05f08df" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:02:20.644069 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:20.643868 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf"] Oct 30 20:02:20.644574 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:20.644160 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" podUID="89b442c9-df9e-42fb-a944-5b1ac933efa4" containerName="maas-api" containerID="cri-o://954651eb219e4a8260454c4cf9275a13a926b58e42b3f0d6bee467ada055b146" gracePeriod=30 Oct 30 20:02:20.942752 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:20.942725 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 20:02:20.988839 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:20.988803 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b442c9-df9e-42fb-a944-5b1ac933efa4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b442c9-df9e-42fb-a944-5b1ac933efa4-openshift-service-ca\") pod \"89b442c9-df9e-42fb-a944-5b1ac933efa4\" (UID: \"89b442c9-df9e-42fb-a944-5b1ac933efa4\") " Oct 30 20:02:21.004220 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:20.988949 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b442c9-df9e-42fb-a944-5b1ac933efa4-kube-api-access-plqpw\" (UniqueName: \"kubernetes.io/projected/89b442c9-df9e-42fb-a944-5b1ac933efa4-kube-api-access-plqpw\") pod \"89b442c9-df9e-42fb-a944-5b1ac933efa4\" (UID: \"89b442c9-df9e-42fb-a944-5b1ac933efa4\") " Oct 30 20:02:21.004220 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:20.988992 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b442c9-df9e-42fb-a944-5b1ac933efa4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b442c9-df9e-42fb-a944-5b1ac933efa4-metrics-certs\") pod \"89b442c9-df9e-42fb-a944-5b1ac933efa4\" (UID: \"89b442c9-df9e-42fb-a944-5b1ac933efa4\") " Oct 30 20:02:21.004220 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:20.989186 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b442c9-df9e-42fb-a944-5b1ac933efa4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b442c9-df9e-42fb-a944-5b1ac933efa4-maas-api-tls\") pod \"89b442c9-df9e-42fb-a944-5b1ac933efa4\" (UID: \"89b442c9-df9e-42fb-a944-5b1ac933efa4\") " Oct 30 20:02:21.004220 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:20.996377 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b442c9-df9e-42fb-a944-5b1ac933efa4-openshift-service-ca" pod "89b442c9-df9e-42fb-a944-5b1ac933efa4" (UID: "89b442c9-df9e-42fb-a944-5b1ac933efa4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:21.010339 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:21.010308 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b442c9-df9e-42fb-a944-5b1ac933efa4-metrics-certs" pod "89b442c9-df9e-42fb-a944-5b1ac933efa4" (UID: "89b442c9-df9e-42fb-a944-5b1ac933efa4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:21.014919 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:21.014887 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b442c9-df9e-42fb-a944-5b1ac933efa4-maas-api-tls" pod "89b442c9-df9e-42fb-a944-5b1ac933efa4" (UID: "89b442c9-df9e-42fb-a944-5b1ac933efa4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:21.015426 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:21.015396 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b442c9-df9e-42fb-a944-5b1ac933efa4-kube-api-access-plqpw" pod "89b442c9-df9e-42fb-a944-5b1ac933efa4" (UID: "89b442c9-df9e-42fb-a944-5b1ac933efa4"). InnerVolumeSpecName "kube-api-access-plqpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:21.092044 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:21.091344 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b442c9-df9e-42fb-a944-5b1ac933efa4-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:21.092044 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:21.091379 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b442c9-df9e-42fb-a944-5b1ac933efa4-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:21.092044 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:21.091396 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plqpw\" (UniqueName: \"kubernetes.io/projected/89b442c9-df9e-42fb-a944-5b1ac933efa4-kube-api-access-plqpw\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:21.092044 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:21.091412 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b442c9-df9e-42fb-a944-5b1ac933efa4-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:21.496080 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:21.496041 5047 generic.go:356] "Generic (PLEG): container finished" podID="89b442c9-df9e-42fb-a944-5b1ac933efa4" containerID="954651eb219e4a8260454c4cf9275a13a926b58e42b3f0d6bee467ada055b146" exitCode=0 Oct 30 20:02:21.496266 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:21.496084 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" event={"ID":"89b442c9-df9e-42fb-a944-5b1ac933efa4","Type":"ContainerDied","Data":"954651eb219e4a8260454c4cf9275a13a926b58e42b3f0d6bee467ada055b146"} Oct 30 20:02:21.496266 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:21.496141 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" event={"ID":"89b442c9-df9e-42fb-a944-5b1ac933efa4","Type":"ContainerDied","Data":"d6c5ef521bb1ecef7c7b6b518db359fd715d57ed72d74f45e16710263cce42d3"} Oct 30 20:02:21.496266 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:21.496165 5047 scope.go:122] "RemoveContainer" containerID="954651eb219e4a8260454c4cf9275a13a926b58e42b3f0d6bee467ada055b146" Oct 30 20:02:21.496266 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:21.496102 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf" Oct 30 20:02:21.506720 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:21.506695 5047 scope.go:122] "RemoveContainer" containerID="954651eb219e4a8260454c4cf9275a13a926b58e42b3f0d6bee467ada055b146" Oct 30 20:02:21.506998 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:02:21.506979 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"954651eb219e4a8260454c4cf9275a13a926b58e42b3f0d6bee467ada055b146\": container with ID starting with 954651eb219e4a8260454c4cf9275a13a926b58e42b3f0d6bee467ada055b146 not found: ID does not exist" containerID="954651eb219e4a8260454c4cf9275a13a926b58e42b3f0d6bee467ada055b146" Oct 30 20:02:21.507121 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:21.507035 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954651eb219e4a8260454c4cf9275a13a926b58e42b3f0d6bee467ada055b146"} err="failed to get container status \"954651eb219e4a8260454c4cf9275a13a926b58e42b3f0d6bee467ada055b146\": rpc error: code = NotFound desc = could not find container \"954651eb219e4a8260454c4cf9275a13a926b58e42b3f0d6bee467ada055b146\": container with ID starting with 954651eb219e4a8260454c4cf9275a13a926b58e42b3f0d6bee467ada055b146 not found: ID does not exist" Oct 30 20:02:21.541705 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:21.541684 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf"] Oct 30 20:02:21.551236 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:21.551213 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-287ea2-75f5b7d465-4xvqf"] Oct 30 20:02:22.488773 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:22.488724 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" podUID="87b98550-4b44-4b24-8c55-03a8e05f08df" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:02:22.489252 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:22.488801 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" podUID="d1e68671-670d-4915-ba4d-47c184994e6a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:02:23.049163 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:23.049127 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b442c9-df9e-42fb-a944-5b1ac933efa4" path="/var/lib/kubelet/pods/89b442c9-df9e-42fb-a944-5b1ac933efa4/volumes" Oct 30 20:02:26.207522 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.207489 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz"] Oct 30 20:02:26.207977 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.207680 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" podUID="64a39903-20d2-4c30-88a0-57cf7c222212" containerName="payload-processing" containerID="cri-o://96a79a1b5b747fe61faaf131b7199e38e1edb9731cb4a930ddd044e314e0811c" gracePeriod=30 Oct 30 20:02:26.232844 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.232818 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4"] Oct 30 20:02:26.233134 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.233097 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" podUID="aa3c8991-f71f-4500-b4b7-e25088017a29" containerName="payload-pre-processing" containerID="cri-o://4d26713405d19b3fbaac0a062d7077ed5a4ad6a30c76f2e3f0bff2e394ae8edc" gracePeriod=30 Oct 30 20:02:26.608100 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.608068 5047 generic.go:356] "Generic (PLEG): container finished" podID="aa3c8991-f71f-4500-b4b7-e25088017a29" containerID="4d26713405d19b3fbaac0a062d7077ed5a4ad6a30c76f2e3f0bff2e394ae8edc" exitCode=0 Oct 30 20:02:26.608239 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.608146 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" event={"ID":"aa3c8991-f71f-4500-b4b7-e25088017a29","Type":"ContainerDied","Data":"4d26713405d19b3fbaac0a062d7077ed5a4ad6a30c76f2e3f0bff2e394ae8edc"} Oct 30 20:02:26.610325 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.610302 5047 generic.go:356] "Generic (PLEG): container finished" podID="64a39903-20d2-4c30-88a0-57cf7c222212" containerID="96a79a1b5b747fe61faaf131b7199e38e1edb9731cb4a930ddd044e314e0811c" exitCode=0 Oct 30 20:02:26.610401 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.610359 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" event={"ID":"64a39903-20d2-4c30-88a0-57cf7c222212","Type":"ContainerDied","Data":"96a79a1b5b747fe61faaf131b7199e38e1edb9731cb4a930ddd044e314e0811c"} Oct 30 20:02:26.612694 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.612680 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" Oct 30 20:02:26.662958 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.662928 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" Oct 30 20:02:26.681402 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.681373 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a39903-20d2-4c30-88a0-57cf7c222212-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a39903-20d2-4c30-88a0-57cf7c222212-plugins-config-volume\") pod \"64a39903-20d2-4c30-88a0-57cf7c222212\" (UID: \"64a39903-20d2-4c30-88a0-57cf7c222212\") " Oct 30 20:02:26.681624 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.681599 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a39903-20d2-4c30-88a0-57cf7c222212-kube-api-access-7rcz8\" (UniqueName: \"kubernetes.io/projected/64a39903-20d2-4c30-88a0-57cf7c222212-kube-api-access-7rcz8\") pod \"64a39903-20d2-4c30-88a0-57cf7c222212\" (UID: \"64a39903-20d2-4c30-88a0-57cf7c222212\") " Oct 30 20:02:26.681799 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.681767 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a39903-20d2-4c30-88a0-57cf7c222212-plugins-config-volume" pod "64a39903-20d2-4c30-88a0-57cf7c222212" (UID: "64a39903-20d2-4c30-88a0-57cf7c222212"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:26.682156 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.682135 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64a39903-20d2-4c30-88a0-57cf7c222212-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:26.684356 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.684331 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a39903-20d2-4c30-88a0-57cf7c222212-kube-api-access-7rcz8" pod "64a39903-20d2-4c30-88a0-57cf7c222212" (UID: "64a39903-20d2-4c30-88a0-57cf7c222212"). InnerVolumeSpecName "kube-api-access-7rcz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:26.783533 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.783455 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa3c8991-f71f-4500-b4b7-e25088017a29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3c8991-f71f-4500-b4b7-e25088017a29-plugins-config-volume\") pod \"aa3c8991-f71f-4500-b4b7-e25088017a29\" (UID: \"aa3c8991-f71f-4500-b4b7-e25088017a29\") " Oct 30 20:02:26.783533 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.783510 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa3c8991-f71f-4500-b4b7-e25088017a29-kube-api-access-8pjnj\" (UniqueName: \"kubernetes.io/projected/aa3c8991-f71f-4500-b4b7-e25088017a29-kube-api-access-8pjnj\") pod \"aa3c8991-f71f-4500-b4b7-e25088017a29\" (UID: \"aa3c8991-f71f-4500-b4b7-e25088017a29\") " Oct 30 20:02:26.783815 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.783793 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa3c8991-f71f-4500-b4b7-e25088017a29-plugins-config-volume" pod "aa3c8991-f71f-4500-b4b7-e25088017a29" (UID: "aa3c8991-f71f-4500-b4b7-e25088017a29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:26.783982 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.783967 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rcz8\" (UniqueName: \"kubernetes.io/projected/64a39903-20d2-4c30-88a0-57cf7c222212-kube-api-access-7rcz8\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:26.784072 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.783984 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3c8991-f71f-4500-b4b7-e25088017a29-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:26.785678 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.785649 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3c8991-f71f-4500-b4b7-e25088017a29-kube-api-access-8pjnj" pod "aa3c8991-f71f-4500-b4b7-e25088017a29" (UID: "aa3c8991-f71f-4500-b4b7-e25088017a29"). InnerVolumeSpecName "kube-api-access-8pjnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:26.885612 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:26.885579 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pjnj\" (UniqueName: \"kubernetes.io/projected/aa3c8991-f71f-4500-b4b7-e25088017a29-kube-api-access-8pjnj\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:27.622282 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:27.622248 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" Oct 30 20:02:27.622282 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:27.622267 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz" event={"ID":"64a39903-20d2-4c30-88a0-57cf7c222212","Type":"ContainerDied","Data":"b42247c645426bfcd03e8465dddcc04ca80c28a41c412b742dc4fa4ec5a24078"} Oct 30 20:02:27.622790 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:27.622344 5047 scope.go:122] "RemoveContainer" containerID="96a79a1b5b747fe61faaf131b7199e38e1edb9731cb4a930ddd044e314e0811c" Oct 30 20:02:27.623850 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:27.623830 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" event={"ID":"aa3c8991-f71f-4500-b4b7-e25088017a29","Type":"ContainerDied","Data":"69fae77aeecfef553f19914abfc213b75941a0feefda727eba524164c6436cbf"} Oct 30 20:02:27.623943 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:27.623855 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4" Oct 30 20:02:27.632554 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:27.632536 5047 scope.go:122] "RemoveContainer" containerID="4d26713405d19b3fbaac0a062d7077ed5a4ad6a30c76f2e3f0bff2e394ae8edc" Oct 30 20:02:27.668137 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:27.668108 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4"] Oct 30 20:02:27.672272 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:27.672251 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fe3e5e-549874cbf5-42bf4"] Oct 30 20:02:27.700998 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:27.700975 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz"] Oct 30 20:02:27.706099 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:27.706080 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fe3e5e-78f5f9954d-n8qxz"] Oct 30 20:02:29.050417 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:29.050382 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a39903-20d2-4c30-88a0-57cf7c222212" path="/var/lib/kubelet/pods/64a39903-20d2-4c30-88a0-57cf7c222212/volumes" Oct 30 20:02:29.050831 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:29.050817 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa3c8991-f71f-4500-b4b7-e25088017a29" path="/var/lib/kubelet/pods/aa3c8991-f71f-4500-b4b7-e25088017a29/volumes" Oct 30 20:02:30.346374 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.346349 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" Oct 30 20:02:30.367438 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.367419 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" Oct 30 20:02:30.427782 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.427743 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87b98550-4b44-4b24-8c55-03a8e05f08df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b98550-4b44-4b24-8c55-03a8e05f08df-plugins-config-volume\") pod \"87b98550-4b44-4b24-8c55-03a8e05f08df\" (UID: \"87b98550-4b44-4b24-8c55-03a8e05f08df\") " Oct 30 20:02:30.427950 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.427830 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b98550-4b44-4b24-8c55-03a8e05f08df-kube-api-access-fdpx8\" (UniqueName: \"kubernetes.io/projected/87b98550-4b44-4b24-8c55-03a8e05f08df-kube-api-access-fdpx8\") pod \"87b98550-4b44-4b24-8c55-03a8e05f08df\" (UID: \"87b98550-4b44-4b24-8c55-03a8e05f08df\") " Oct 30 20:02:30.428221 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.428192 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b98550-4b44-4b24-8c55-03a8e05f08df-plugins-config-volume" pod "87b98550-4b44-4b24-8c55-03a8e05f08df" (UID: "87b98550-4b44-4b24-8c55-03a8e05f08df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:30.428519 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.428498 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b98550-4b44-4b24-8c55-03a8e05f08df-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:30.429996 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.429972 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b98550-4b44-4b24-8c55-03a8e05f08df-kube-api-access-fdpx8" pod "87b98550-4b44-4b24-8c55-03a8e05f08df" (UID: "87b98550-4b44-4b24-8c55-03a8e05f08df"). InnerVolumeSpecName "kube-api-access-fdpx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:30.529282 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.529241 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e68671-670d-4915-ba4d-47c184994e6a-kube-api-access-j9zwx\" (UniqueName: \"kubernetes.io/projected/d1e68671-670d-4915-ba4d-47c184994e6a-kube-api-access-j9zwx\") pod \"d1e68671-670d-4915-ba4d-47c184994e6a\" (UID: \"d1e68671-670d-4915-ba4d-47c184994e6a\") " Oct 30 20:02:30.529464 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.529426 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e68671-670d-4915-ba4d-47c184994e6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e68671-670d-4915-ba4d-47c184994e6a-plugins-config-volume\") pod \"d1e68671-670d-4915-ba4d-47c184994e6a\" (UID: \"d1e68671-670d-4915-ba4d-47c184994e6a\") " Oct 30 20:02:30.529767 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.529745 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e68671-670d-4915-ba4d-47c184994e6a-plugins-config-volume" pod "d1e68671-670d-4915-ba4d-47c184994e6a" (UID: "d1e68671-670d-4915-ba4d-47c184994e6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:30.529906 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.529887 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdpx8\" (UniqueName: \"kubernetes.io/projected/87b98550-4b44-4b24-8c55-03a8e05f08df-kube-api-access-fdpx8\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:30.529952 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.529913 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e68671-670d-4915-ba4d-47c184994e6a-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:30.531431 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.531408 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e68671-670d-4915-ba4d-47c184994e6a-kube-api-access-j9zwx" pod "d1e68671-670d-4915-ba4d-47c184994e6a" (UID: "d1e68671-670d-4915-ba4d-47c184994e6a"). InnerVolumeSpecName "kube-api-access-j9zwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:30.630713 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.630681 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9zwx\" (UniqueName: \"kubernetes.io/projected/d1e68671-670d-4915-ba4d-47c184994e6a-kube-api-access-j9zwx\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:30.646606 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.646577 5047 generic.go:356] "Generic (PLEG): container finished" podID="87b98550-4b44-4b24-8c55-03a8e05f08df" containerID="86c62f6d2eca6d96ee48d9d091102e0b2eb6fe3e08c3ac42120a954967183d47" exitCode=0 Oct 30 20:02:30.646784 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.646624 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" Oct 30 20:02:30.646784 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.646655 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" event={"ID":"87b98550-4b44-4b24-8c55-03a8e05f08df","Type":"ContainerDied","Data":"86c62f6d2eca6d96ee48d9d091102e0b2eb6fe3e08c3ac42120a954967183d47"} Oct 30 20:02:30.646784 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.646711 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk" event={"ID":"87b98550-4b44-4b24-8c55-03a8e05f08df","Type":"ContainerDied","Data":"b2f7818da9d491216dd195abf21a2c8074446aee7bf48c76d53044b89f70c215"} Oct 30 20:02:30.646784 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.646740 5047 scope.go:122] "RemoveContainer" containerID="86c62f6d2eca6d96ee48d9d091102e0b2eb6fe3e08c3ac42120a954967183d47" Oct 30 20:02:30.648238 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.648208 5047 generic.go:356] "Generic (PLEG): container finished" podID="d1e68671-670d-4915-ba4d-47c184994e6a" containerID="96c13429ba605e3212ef6e9f50cd199a7d08a999b13b2b7102abe809e2d0683d" exitCode=0 Oct 30 20:02:30.648375 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.648246 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" Oct 30 20:02:30.648449 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.648370 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" event={"ID":"d1e68671-670d-4915-ba4d-47c184994e6a","Type":"ContainerDied","Data":"96c13429ba605e3212ef6e9f50cd199a7d08a999b13b2b7102abe809e2d0683d"} Oct 30 20:02:30.648449 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.648436 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45" event={"ID":"d1e68671-670d-4915-ba4d-47c184994e6a","Type":"ContainerDied","Data":"434bc3c8f77b46f455ace59978f9fbf80919f721060c891f22169ebaa0015637"} Oct 30 20:02:30.658934 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.658917 5047 scope.go:122] "RemoveContainer" containerID="86c62f6d2eca6d96ee48d9d091102e0b2eb6fe3e08c3ac42120a954967183d47" Oct 30 20:02:30.659256 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:02:30.659234 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c62f6d2eca6d96ee48d9d091102e0b2eb6fe3e08c3ac42120a954967183d47\": container with ID starting with 86c62f6d2eca6d96ee48d9d091102e0b2eb6fe3e08c3ac42120a954967183d47 not found: ID does not exist" containerID="86c62f6d2eca6d96ee48d9d091102e0b2eb6fe3e08c3ac42120a954967183d47" Oct 30 20:02:30.659346 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.659263 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c62f6d2eca6d96ee48d9d091102e0b2eb6fe3e08c3ac42120a954967183d47"} err="failed to get container status \"86c62f6d2eca6d96ee48d9d091102e0b2eb6fe3e08c3ac42120a954967183d47\": rpc error: code = NotFound desc = could not find container \"86c62f6d2eca6d96ee48d9d091102e0b2eb6fe3e08c3ac42120a954967183d47\": container with ID starting with 86c62f6d2eca6d96ee48d9d091102e0b2eb6fe3e08c3ac42120a954967183d47 not found: ID does not exist" Oct 30 20:02:30.659346 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.659282 5047 scope.go:122] "RemoveContainer" containerID="96c13429ba605e3212ef6e9f50cd199a7d08a999b13b2b7102abe809e2d0683d" Oct 30 20:02:30.668482 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.668459 5047 scope.go:122] "RemoveContainer" containerID="96c13429ba605e3212ef6e9f50cd199a7d08a999b13b2b7102abe809e2d0683d" Oct 30 20:02:30.668734 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:02:30.668715 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c13429ba605e3212ef6e9f50cd199a7d08a999b13b2b7102abe809e2d0683d\": container with ID starting with 96c13429ba605e3212ef6e9f50cd199a7d08a999b13b2b7102abe809e2d0683d not found: ID does not exist" containerID="96c13429ba605e3212ef6e9f50cd199a7d08a999b13b2b7102abe809e2d0683d" Oct 30 20:02:30.668793 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.668742 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c13429ba605e3212ef6e9f50cd199a7d08a999b13b2b7102abe809e2d0683d"} err="failed to get container status \"96c13429ba605e3212ef6e9f50cd199a7d08a999b13b2b7102abe809e2d0683d\": rpc error: code = NotFound desc = could not find container \"96c13429ba605e3212ef6e9f50cd199a7d08a999b13b2b7102abe809e2d0683d\": container with ID starting with 96c13429ba605e3212ef6e9f50cd199a7d08a999b13b2b7102abe809e2d0683d not found: ID does not exist" Oct 30 20:02:30.704784 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.704753 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk"] Oct 30 20:02:30.714554 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.714528 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b7ccc-86888dcbc6-t29sk"] Oct 30 20:02:30.743329 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.743303 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45"] Oct 30 20:02:30.749922 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:30.749886 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b7ccc-6c54f46899-gcf45"] Oct 30 20:02:31.050762 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:31.050691 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87b98550-4b44-4b24-8c55-03a8e05f08df" path="/var/lib/kubelet/pods/87b98550-4b44-4b24-8c55-03a8e05f08df/volumes" Oct 30 20:02:31.051144 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:31.051129 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e68671-670d-4915-ba4d-47c184994e6a" path="/var/lib/kubelet/pods/d1e68671-670d-4915-ba4d-47c184994e6a/volumes" Oct 30 20:02:36.309335 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:36.309291 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq"] Oct 30 20:02:37.416235 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.416193 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq"] Oct 30 20:02:37.416743 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.416478 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" podUID="ce2235f8-00fb-44bc-90f8-6629cfc22510" containerName="maas-api" containerID="cri-o://2ccd7acd3c6787922296acfdf072bf33e68ec34f2d58c294ba41cd7a6de14637" gracePeriod=30 Oct 30 20:02:37.488822 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.488774 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" podUID="ce2235f8-00fb-44bc-90f8-6629cfc22510" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.220:8443/health\": dial tcp 10.132.0.220:8443: connect: connection refused" Oct 30 20:02:37.695205 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.695175 5047 generic.go:356] "Generic (PLEG): container finished" podID="ce2235f8-00fb-44bc-90f8-6629cfc22510" containerID="2ccd7acd3c6787922296acfdf072bf33e68ec34f2d58c294ba41cd7a6de14637" exitCode=0 Oct 30 20:02:37.695351 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.695246 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" event={"ID":"ce2235f8-00fb-44bc-90f8-6629cfc22510","Type":"ContainerDied","Data":"2ccd7acd3c6787922296acfdf072bf33e68ec34f2d58c294ba41cd7a6de14637"} Oct 30 20:02:37.695351 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.695344 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" event={"ID":"ce2235f8-00fb-44bc-90f8-6629cfc22510","Type":"ContainerDied","Data":"ef97b891d640f370c85f05e7604bd68023bf7cfbac46bb760b5bf2e34c88e38f"} Oct 30 20:02:37.695444 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.695366 5047 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef97b891d640f370c85f05e7604bd68023bf7cfbac46bb760b5bf2e34c88e38f" Oct 30 20:02:37.703051 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.703024 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 20:02:37.722401 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.722375 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce2235f8-00fb-44bc-90f8-6629cfc22510-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce2235f8-00fb-44bc-90f8-6629cfc22510-metrics-certs\") pod \"ce2235f8-00fb-44bc-90f8-6629cfc22510\" (UID: \"ce2235f8-00fb-44bc-90f8-6629cfc22510\") " Oct 30 20:02:37.722509 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.722448 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce2235f8-00fb-44bc-90f8-6629cfc22510-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce2235f8-00fb-44bc-90f8-6629cfc22510-maas-api-tls\") pod \"ce2235f8-00fb-44bc-90f8-6629cfc22510\" (UID: \"ce2235f8-00fb-44bc-90f8-6629cfc22510\") " Oct 30 20:02:37.722509 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.722497 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce2235f8-00fb-44bc-90f8-6629cfc22510-kube-api-access-nrcsx\" (UniqueName: \"kubernetes.io/projected/ce2235f8-00fb-44bc-90f8-6629cfc22510-kube-api-access-nrcsx\") pod \"ce2235f8-00fb-44bc-90f8-6629cfc22510\" (UID: \"ce2235f8-00fb-44bc-90f8-6629cfc22510\") " Oct 30 20:02:37.722596 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.722527 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce2235f8-00fb-44bc-90f8-6629cfc22510-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2235f8-00fb-44bc-90f8-6629cfc22510-openshift-service-ca\") pod \"ce2235f8-00fb-44bc-90f8-6629cfc22510\" (UID: \"ce2235f8-00fb-44bc-90f8-6629cfc22510\") " Oct 30 20:02:37.722947 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.722922 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2235f8-00fb-44bc-90f8-6629cfc22510-openshift-service-ca" pod "ce2235f8-00fb-44bc-90f8-6629cfc22510" (UID: "ce2235f8-00fb-44bc-90f8-6629cfc22510"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:37.724521 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.724500 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce2235f8-00fb-44bc-90f8-6629cfc22510-maas-api-tls" pod "ce2235f8-00fb-44bc-90f8-6629cfc22510" (UID: "ce2235f8-00fb-44bc-90f8-6629cfc22510"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:37.724636 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.724565 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2235f8-00fb-44bc-90f8-6629cfc22510-kube-api-access-nrcsx" pod "ce2235f8-00fb-44bc-90f8-6629cfc22510" (UID: "ce2235f8-00fb-44bc-90f8-6629cfc22510"). InnerVolumeSpecName "kube-api-access-nrcsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:37.724636 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.724620 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce2235f8-00fb-44bc-90f8-6629cfc22510-metrics-certs" pod "ce2235f8-00fb-44bc-90f8-6629cfc22510" (UID: "ce2235f8-00fb-44bc-90f8-6629cfc22510"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:37.824267 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.824237 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce2235f8-00fb-44bc-90f8-6629cfc22510-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:37.824267 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.824263 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrcsx\" (UniqueName: \"kubernetes.io/projected/ce2235f8-00fb-44bc-90f8-6629cfc22510-kube-api-access-nrcsx\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:37.824267 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.824274 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2235f8-00fb-44bc-90f8-6629cfc22510-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:37.824493 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:37.824282 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce2235f8-00fb-44bc-90f8-6629cfc22510-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:02:38.702434 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:38.702404 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq" Oct 30 20:02:38.749189 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:38.749149 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq"] Oct 30 20:02:38.756972 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:38.756940 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e430a0-77755b6b4f-6zftq"] Oct 30 20:02:39.049352 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:39.049273 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce2235f8-00fb-44bc-90f8-6629cfc22510" path="/var/lib/kubelet/pods/ce2235f8-00fb-44bc-90f8-6629cfc22510/volumes" Oct 30 20:02:39.600979 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:02:39.600934 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:02:41.309902 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:41.309856 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" podUID="4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0" containerName="payload-pre-processing" containerID="cri-o://3928d8b75b01c118473b9989d7b2dddf74eec63440af3c0643f278b8dd4df711" gracePeriod=25 Oct 30 20:02:51.093784 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:02:51.093730 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" podUID="4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:01.093649 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.093607 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" podUID="4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:01.633589 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.633556 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" Oct 30 20:03:01.694355 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.694319 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt"] Oct 30 20:03:01.694595 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.694571 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" podUID="bfef2562-28c6-4640-a1c0-8a467bba8a79" containerName="istio-proxy" containerID="cri-o://5c6cea449c1c2678df502ac0ed48cc5fe496d47722d4f18429973ea3e2107f3e" gracePeriod=30 Oct 30 20:03:01.708383 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.708357 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt"] Oct 30 20:03:01.708565 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.708522 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" podUID="b913225f-a9cf-4af0-af6f-076a9c0ec683" containerName="istio-proxy" containerID="cri-o://8b40dc18b077766fa4a849a56062cf6dc7ea6135eb0969a71834480c484d80c5" gracePeriod=30 Oct 30 20:03:01.756849 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.756813 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0-kube-api-access-wbgqn\" (UniqueName: \"kubernetes.io/projected/4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0-kube-api-access-wbgqn\") pod \"4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0\" (UID: \"4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0\") " Oct 30 20:03:01.757081 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.756995 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0-plugins-config-volume\") pod \"4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0\" (UID: \"4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0\") " Oct 30 20:03:01.757951 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.757921 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0-plugins-config-volume" pod "4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0" (UID: "4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:01.760427 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.760397 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0-kube-api-access-wbgqn" pod "4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0" (UID: "4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0"). InnerVolumeSpecName "kube-api-access-wbgqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:01.870516 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.867077 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:01.870516 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.867112 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbgqn\" (UniqueName: \"kubernetes.io/projected/4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0-kube-api-access-wbgqn\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:01.891536 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.891464 5047 generic.go:356] "Generic (PLEG): container finished" podID="4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0" containerID="3928d8b75b01c118473b9989d7b2dddf74eec63440af3c0643f278b8dd4df711" exitCode=0 Oct 30 20:03:01.891830 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.891808 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" event={"ID":"4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0","Type":"ContainerDied","Data":"3928d8b75b01c118473b9989d7b2dddf74eec63440af3c0643f278b8dd4df711"} Oct 30 20:03:01.891931 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.891886 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" event={"ID":"4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0","Type":"ContainerDied","Data":"c4a40bf35c96dfd70305d56d6303697f8647e74dd01542411c3238e1f9594e3d"} Oct 30 20:03:01.891993 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.891939 5047 scope.go:122] "RemoveContainer" containerID="3928d8b75b01c118473b9989d7b2dddf74eec63440af3c0643f278b8dd4df711" Oct 30 20:03:01.898677 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.894339 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq" Oct 30 20:03:01.907557 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.907536 5047 scope.go:122] "RemoveContainer" containerID="3928d8b75b01c118473b9989d7b2dddf74eec63440af3c0643f278b8dd4df711" Oct 30 20:03:01.907894 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:03:01.907868 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3928d8b75b01c118473b9989d7b2dddf74eec63440af3c0643f278b8dd4df711\": container with ID starting with 3928d8b75b01c118473b9989d7b2dddf74eec63440af3c0643f278b8dd4df711 not found: ID does not exist" containerID="3928d8b75b01c118473b9989d7b2dddf74eec63440af3c0643f278b8dd4df711" Oct 30 20:03:01.907982 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.907901 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3928d8b75b01c118473b9989d7b2dddf74eec63440af3c0643f278b8dd4df711"} err="failed to get container status \"3928d8b75b01c118473b9989d7b2dddf74eec63440af3c0643f278b8dd4df711\": rpc error: code = NotFound desc = could not find container \"3928d8b75b01c118473b9989d7b2dddf74eec63440af3c0643f278b8dd4df711\": container with ID starting with 3928d8b75b01c118473b9989d7b2dddf74eec63440af3c0643f278b8dd4df711 not found: ID does not exist" Oct 30 20:03:01.939972 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.939940 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq"] Oct 30 20:03:01.944346 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:01.944319 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9eb76-57bb6d59f5-b7rdq"] Oct 30 20:03:03.048494 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.048459 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0" path="/var/lib/kubelet/pods/4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0/volumes" Oct 30 20:03:03.134756 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.134719 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:03:03.134937 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.134778 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" podUID="bfef2562-28c6-4640-a1c0-8a467bba8a79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:03:03.424175 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.424136 5047 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:03:03.424355 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.424211 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" podUID="b913225f-a9cf-4af0-af6f-076a9c0ec683" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:03:03.435701 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.435668 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t"] Oct 30 20:03:03.435904 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.435882 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" podUID="52b11843-e441-4d53-a386-bc1b63404d00" containerName="maas-api" containerID="cri-o://7a03210aca63a65f695ee7fad88e85874a53373faeb0611044d40e7c4f3f2400" gracePeriod=30 Oct 30 20:03:03.600751 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.600715 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff"] Oct 30 20:03:03.600954 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.600923 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff" podUID="2ac686d8-338e-43b4-81b0-15718ed61608" containerName="payload-processing" containerID="cri-o://37fff8009a6251d4f9224ee1423aec63b3be3228dff3b9255f4ecb26b3aa5e7a" gracePeriod=30 Oct 30 20:03:03.630989 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.628530 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g"] Oct 30 20:03:03.630989 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.628830 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" podUID="845e76e9-51c7-440d-a89e-60adc522d008" containerName="payload-pre-processing" containerID="cri-o://b21cdb35f6eb3f1d36fc5eb18699fef34297a35d5692aba97ed9043bb418847a" gracePeriod=30 Oct 30 20:03:03.769614 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.769588 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:03:03.895351 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.895315 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52b11843-e441-4d53-a386-bc1b63404d00-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b11843-e441-4d53-a386-bc1b63404d00-metrics-certs\") pod \"52b11843-e441-4d53-a386-bc1b63404d00\" (UID: \"52b11843-e441-4d53-a386-bc1b63404d00\") " Oct 30 20:03:03.895519 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.895447 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52b11843-e441-4d53-a386-bc1b63404d00-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52b11843-e441-4d53-a386-bc1b63404d00-maas-api-tls\") pod \"52b11843-e441-4d53-a386-bc1b63404d00\" (UID: \"52b11843-e441-4d53-a386-bc1b63404d00\") " Oct 30 20:03:03.895584 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.895538 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b11843-e441-4d53-a386-bc1b63404d00-kube-api-access-cvpg7\" (UniqueName: \"kubernetes.io/projected/52b11843-e441-4d53-a386-bc1b63404d00-kube-api-access-cvpg7\") pod \"52b11843-e441-4d53-a386-bc1b63404d00\" (UID: \"52b11843-e441-4d53-a386-bc1b63404d00\") " Oct 30 20:03:03.895649 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.895632 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b11843-e441-4d53-a386-bc1b63404d00-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b11843-e441-4d53-a386-bc1b63404d00-openshift-service-ca\") pod \"52b11843-e441-4d53-a386-bc1b63404d00\" (UID: \"52b11843-e441-4d53-a386-bc1b63404d00\") " Oct 30 20:03:03.897189 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.896826 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b11843-e441-4d53-a386-bc1b63404d00-openshift-service-ca" pod "52b11843-e441-4d53-a386-bc1b63404d00" (UID: "52b11843-e441-4d53-a386-bc1b63404d00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:03.897780 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.897754 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b11843-e441-4d53-a386-bc1b63404d00-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:03.900493 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.900455 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52b11843-e441-4d53-a386-bc1b63404d00-maas-api-tls" pod "52b11843-e441-4d53-a386-bc1b63404d00" (UID: "52b11843-e441-4d53-a386-bc1b63404d00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:03.900642 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.900485 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52b11843-e441-4d53-a386-bc1b63404d00-metrics-certs" pod "52b11843-e441-4d53-a386-bc1b63404d00" (UID: "52b11843-e441-4d53-a386-bc1b63404d00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:03.906266 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.905989 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b11843-e441-4d53-a386-bc1b63404d00-kube-api-access-cvpg7" pod "52b11843-e441-4d53-a386-bc1b63404d00" (UID: "52b11843-e441-4d53-a386-bc1b63404d00"). InnerVolumeSpecName "kube-api-access-cvpg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:03.923633 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.923597 5047 generic.go:356] "Generic (PLEG): container finished" podID="52b11843-e441-4d53-a386-bc1b63404d00" containerID="7a03210aca63a65f695ee7fad88e85874a53373faeb0611044d40e7c4f3f2400" exitCode=0 Oct 30 20:03:03.924878 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.923990 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" Oct 30 20:03:03.931272 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.924060 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" event={"ID":"52b11843-e441-4d53-a386-bc1b63404d00","Type":"ContainerDied","Data":"7a03210aca63a65f695ee7fad88e85874a53373faeb0611044d40e7c4f3f2400"} Oct 30 20:03:03.931664 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.931606 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t" event={"ID":"52b11843-e441-4d53-a386-bc1b63404d00","Type":"ContainerDied","Data":"22f2fc9e353fa171e99e596261a587599c4d37dbd8d24122e2288b179f77fb9b"} Oct 30 20:03:03.931850 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.931779 5047 scope.go:122] "RemoveContainer" containerID="7a03210aca63a65f695ee7fad88e85874a53373faeb0611044d40e7c4f3f2400" Oct 30 20:03:03.970902 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.970866 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff" Oct 30 20:03:03.980359 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.980334 5047 scope.go:122] "RemoveContainer" containerID="7a03210aca63a65f695ee7fad88e85874a53373faeb0611044d40e7c4f3f2400" Oct 30 20:03:03.984756 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:03:03.984717 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a03210aca63a65f695ee7fad88e85874a53373faeb0611044d40e7c4f3f2400\": container with ID starting with 7a03210aca63a65f695ee7fad88e85874a53373faeb0611044d40e7c4f3f2400 not found: ID does not exist" containerID="7a03210aca63a65f695ee7fad88e85874a53373faeb0611044d40e7c4f3f2400" Oct 30 20:03:03.984902 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.984767 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a03210aca63a65f695ee7fad88e85874a53373faeb0611044d40e7c4f3f2400"} err="failed to get container status \"7a03210aca63a65f695ee7fad88e85874a53373faeb0611044d40e7c4f3f2400\": rpc error: code = NotFound desc = could not find container \"7a03210aca63a65f695ee7fad88e85874a53373faeb0611044d40e7c4f3f2400\": container with ID starting with 7a03210aca63a65f695ee7fad88e85874a53373faeb0611044d40e7c4f3f2400 not found: ID does not exist" Oct 30 20:03:03.988299 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.988271 5047 generic.go:356] "Generic (PLEG): container finished" podID="845e76e9-51c7-440d-a89e-60adc522d008" containerID="b21cdb35f6eb3f1d36fc5eb18699fef34297a35d5692aba97ed9043bb418847a" exitCode=0 Oct 30 20:03:03.988435 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.988342 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" event={"ID":"845e76e9-51c7-440d-a89e-60adc522d008","Type":"ContainerDied","Data":"b21cdb35f6eb3f1d36fc5eb18699fef34297a35d5692aba97ed9043bb418847a"} Oct 30 20:03:03.990919 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.990892 5047 generic.go:356] "Generic (PLEG): container finished" podID="2ac686d8-338e-43b4-81b0-15718ed61608" containerID="37fff8009a6251d4f9224ee1423aec63b3be3228dff3b9255f4ecb26b3aa5e7a" exitCode=0 Oct 30 20:03:03.991158 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.990945 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff" Oct 30 20:03:03.991490 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.990968 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff" event={"ID":"2ac686d8-338e-43b4-81b0-15718ed61608","Type":"ContainerDied","Data":"37fff8009a6251d4f9224ee1423aec63b3be3228dff3b9255f4ecb26b3aa5e7a"} Oct 30 20:03:03.991645 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.991631 5047 scope.go:122] "RemoveContainer" containerID="37fff8009a6251d4f9224ee1423aec63b3be3228dff3b9255f4ecb26b3aa5e7a" Oct 30 20:03:03.999328 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.999299 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b11843-e441-4d53-a386-bc1b63404d00-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:03.999435 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.999330 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52b11843-e441-4d53-a386-bc1b63404d00-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:03.999435 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:03.999344 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvpg7\" (UniqueName: \"kubernetes.io/projected/52b11843-e441-4d53-a386-bc1b63404d00-kube-api-access-cvpg7\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:04.000862 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.000833 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" Oct 30 20:03:04.042259 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.042196 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t"] Oct 30 20:03:04.046494 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.046457 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a76aa2-849d65fcb4-mb76t"] Oct 30 20:03:04.100343 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.100308 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845e76e9-51c7-440d-a89e-60adc522d008-kube-api-access-wmfts\" (UniqueName: \"kubernetes.io/projected/845e76e9-51c7-440d-a89e-60adc522d008-kube-api-access-wmfts\") pod \"845e76e9-51c7-440d-a89e-60adc522d008\" (UID: \"845e76e9-51c7-440d-a89e-60adc522d008\") " Oct 30 20:03:04.100895 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.100416 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac686d8-338e-43b4-81b0-15718ed61608-kube-api-access-xx6bc\" (UniqueName: \"kubernetes.io/projected/2ac686d8-338e-43b4-81b0-15718ed61608-kube-api-access-xx6bc\") pod \"2ac686d8-338e-43b4-81b0-15718ed61608\" (UID: \"2ac686d8-338e-43b4-81b0-15718ed61608\") " Oct 30 20:03:04.100895 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.100472 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ac686d8-338e-43b4-81b0-15718ed61608-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac686d8-338e-43b4-81b0-15718ed61608-plugins-config-volume\") pod \"2ac686d8-338e-43b4-81b0-15718ed61608\" (UID: \"2ac686d8-338e-43b4-81b0-15718ed61608\") " Oct 30 20:03:04.100895 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.100575 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/845e76e9-51c7-440d-a89e-60adc522d008-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845e76e9-51c7-440d-a89e-60adc522d008-plugins-config-volume\") pod \"845e76e9-51c7-440d-a89e-60adc522d008\" (UID: \"845e76e9-51c7-440d-a89e-60adc522d008\") " Oct 30 20:03:04.101147 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.100998 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac686d8-338e-43b4-81b0-15718ed61608-plugins-config-volume" pod "2ac686d8-338e-43b4-81b0-15718ed61608" (UID: "2ac686d8-338e-43b4-81b0-15718ed61608"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:04.101212 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.101192 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac686d8-338e-43b4-81b0-15718ed61608-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:04.101520 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.101496 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845e76e9-51c7-440d-a89e-60adc522d008-plugins-config-volume" pod "845e76e9-51c7-440d-a89e-60adc522d008" (UID: "845e76e9-51c7-440d-a89e-60adc522d008"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:04.102851 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.102828 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac686d8-338e-43b4-81b0-15718ed61608-kube-api-access-xx6bc" pod "2ac686d8-338e-43b4-81b0-15718ed61608" (UID: "2ac686d8-338e-43b4-81b0-15718ed61608"). InnerVolumeSpecName "kube-api-access-xx6bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:04.103121 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.103097 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845e76e9-51c7-440d-a89e-60adc522d008-kube-api-access-wmfts" pod "845e76e9-51c7-440d-a89e-60adc522d008" (UID: "845e76e9-51c7-440d-a89e-60adc522d008"). InnerVolumeSpecName "kube-api-access-wmfts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:04.202034 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.201976 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845e76e9-51c7-440d-a89e-60adc522d008-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:04.202034 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.202040 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmfts\" (UniqueName: \"kubernetes.io/projected/845e76e9-51c7-440d-a89e-60adc522d008-kube-api-access-wmfts\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:04.202249 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.202055 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx6bc\" (UniqueName: \"kubernetes.io/projected/2ac686d8-338e-43b4-81b0-15718ed61608-kube-api-access-xx6bc\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:04.337118 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.336984 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff"] Oct 30 20:03:04.339998 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.339973 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a76aa2-7f9fcff8d-qcrff"] Oct 30 20:03:04.999941 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.999906 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" Oct 30 20:03:05.000128 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:04.999906 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g" event={"ID":"845e76e9-51c7-440d-a89e-60adc522d008","Type":"ContainerDied","Data":"8805e0d872a78035b3336c90f30d74d6cbce4e32c1f3a6a0d75840c92f9976f6"} Oct 30 20:03:05.000128 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:05.000055 5047 scope.go:122] "RemoveContainer" containerID="b21cdb35f6eb3f1d36fc5eb18699fef34297a35d5692aba97ed9043bb418847a" Oct 30 20:03:05.050266 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:05.050233 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ac686d8-338e-43b4-81b0-15718ed61608" path="/var/lib/kubelet/pods/2ac686d8-338e-43b4-81b0-15718ed61608/volumes" Oct 30 20:03:05.050690 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:05.050677 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b11843-e441-4d53-a386-bc1b63404d00" path="/var/lib/kubelet/pods/52b11843-e441-4d53-a386-bc1b63404d00/volumes" Oct 30 20:03:05.051086 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:05.051065 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g"] Oct 30 20:03:05.054274 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:05.054257 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a76aa2-76ccb668f9-9bd5g"] Oct 30 20:03:07.019966 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.019936 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 20:03:07.021343 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.021319 5047 generic.go:356] "Generic (PLEG): container finished" podID="bfef2562-28c6-4640-a1c0-8a467bba8a79" containerID="5c6cea449c1c2678df502ac0ed48cc5fe496d47722d4f18429973ea3e2107f3e" exitCode=0 Oct 30 20:03:07.021460 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.021396 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" event={"ID":"bfef2562-28c6-4640-a1c0-8a467bba8a79","Type":"ContainerDied","Data":"5c6cea449c1c2678df502ac0ed48cc5fe496d47722d4f18429973ea3e2107f3e"} Oct 30 20:03:07.021525 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.021464 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" event={"ID":"bfef2562-28c6-4640-a1c0-8a467bba8a79","Type":"ContainerDied","Data":"36e2a53bfc4caeaa8dbcad2569619694e9d9a2a5ad2de498997d85cc29b8ac37"} Oct 30 20:03:07.021571 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.021507 5047 scope.go:122] "RemoveContainer" containerID="5c6cea449c1c2678df502ac0ed48cc5fe496d47722d4f18429973ea3e2107f3e" Oct 30 20:03:07.022833 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.022809 5047 generic.go:356] "Generic (PLEG): container finished" podID="b913225f-a9cf-4af0-af6f-076a9c0ec683" containerID="8b40dc18b077766fa4a849a56062cf6dc7ea6135eb0969a71834480c484d80c5" exitCode=0 Oct 30 20:03:07.022910 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.022874 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" event={"ID":"b913225f-a9cf-4af0-af6f-076a9c0ec683","Type":"ContainerDied","Data":"8b40dc18b077766fa4a849a56062cf6dc7ea6135eb0969a71834480c484d80c5"} Oct 30 20:03:07.044067 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.043998 5047 scope.go:122] "RemoveContainer" containerID="5c6cea449c1c2678df502ac0ed48cc5fe496d47722d4f18429973ea3e2107f3e" Oct 30 20:03:07.044485 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:03:07.044448 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6cea449c1c2678df502ac0ed48cc5fe496d47722d4f18429973ea3e2107f3e\": container with ID starting with 5c6cea449c1c2678df502ac0ed48cc5fe496d47722d4f18429973ea3e2107f3e not found: ID does not exist" containerID="5c6cea449c1c2678df502ac0ed48cc5fe496d47722d4f18429973ea3e2107f3e" Oct 30 20:03:07.044611 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.044499 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6cea449c1c2678df502ac0ed48cc5fe496d47722d4f18429973ea3e2107f3e"} err="failed to get container status \"5c6cea449c1c2678df502ac0ed48cc5fe496d47722d4f18429973ea3e2107f3e\": rpc error: code = NotFound desc = could not find container \"5c6cea449c1c2678df502ac0ed48cc5fe496d47722d4f18429973ea3e2107f3e\": container with ID starting with 5c6cea449c1c2678df502ac0ed48cc5fe496d47722d4f18429973ea3e2107f3e not found: ID does not exist" Oct 30 20:03:07.049399 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.049373 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 20:03:07.052446 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.052415 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845e76e9-51c7-440d-a89e-60adc522d008" path="/var/lib/kubelet/pods/845e76e9-51c7-440d-a89e-60adc522d008/volumes" Oct 30 20:03:07.141847 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.141809 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-workload-socket\") pod \"b913225f-a9cf-4af0-af6f-076a9c0ec683\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " Oct 30 20:03:07.142064 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.141894 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfef2562-28c6-4640-a1c0-8a467bba8a79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfef2562-28c6-4640-a1c0-8a467bba8a79-istiod-ca-cert\") pod \"bfef2562-28c6-4640-a1c0-8a467bba8a79\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " Oct 30 20:03:07.142064 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.141944 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-envoy\") pod \"bfef2562-28c6-4640-a1c0-8a467bba8a79\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " Oct 30 20:03:07.142064 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.141975 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-token\" (UniqueName: \"kubernetes.io/projected/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-token\") pod \"bfef2562-28c6-4640-a1c0-8a467bba8a79\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " Oct 30 20:03:07.142064 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.142024 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-envoy\") pod \"b913225f-a9cf-4af0-af6f-076a9c0ec683\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " Oct 30 20:03:07.142064 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.142056 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-workload-certs\") pod \"b913225f-a9cf-4af0-af6f-076a9c0ec683\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " Oct 30 20:03:07.142370 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.142119 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-podinfo\") pod \"b913225f-a9cf-4af0-af6f-076a9c0ec683\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " Oct 30 20:03:07.142370 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.142168 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b913225f-a9cf-4af0-af6f-076a9c0ec683-kube-api-access-97rkd\" (UniqueName: \"kubernetes.io/projected/b913225f-a9cf-4af0-af6f-076a9c0ec683-kube-api-access-97rkd\") pod \"b913225f-a9cf-4af0-af6f-076a9c0ec683\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " Oct 30 20:03:07.142370 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.142168 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-workload-socket" pod "b913225f-a9cf-4af0-af6f-076a9c0ec683" (UID: "b913225f-a9cf-4af0-af6f-076a9c0ec683"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:07.142370 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.142237 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-data\") pod \"bfef2562-28c6-4640-a1c0-8a467bba8a79\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " Oct 30 20:03:07.142370 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.142275 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-workload-certs\") pod \"bfef2562-28c6-4640-a1c0-8a467bba8a79\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " Oct 30 20:03:07.142370 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.142308 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfef2562-28c6-4640-a1c0-8a467bba8a79-kube-api-access-hffxn\" (UniqueName: \"kubernetes.io/projected/bfef2562-28c6-4640-a1c0-8a467bba8a79-kube-api-access-hffxn\") pod \"bfef2562-28c6-4640-a1c0-8a467bba8a79\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " Oct 30 20:03:07.142370 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.142365 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b913225f-a9cf-4af0-af6f-076a9c0ec683-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b913225f-a9cf-4af0-af6f-076a9c0ec683-istiod-ca-cert\") pod \"b913225f-a9cf-4af0-af6f-076a9c0ec683\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " Oct 30 20:03:07.145334 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.142396 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-credential-socket\") pod \"b913225f-a9cf-4af0-af6f-076a9c0ec683\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " Oct 30 20:03:07.145334 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.142401 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfef2562-28c6-4640-a1c0-8a467bba8a79-istiod-ca-cert" pod "bfef2562-28c6-4640-a1c0-8a467bba8a79" (UID: "bfef2562-28c6-4640-a1c0-8a467bba8a79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:07.145334 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.142426 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-credential-socket\") pod \"bfef2562-28c6-4640-a1c0-8a467bba8a79\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " Oct 30 20:03:07.145334 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.142465 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-podinfo\") pod \"bfef2562-28c6-4640-a1c0-8a467bba8a79\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " Oct 30 20:03:07.145334 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.142499 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-workload-socket\") pod \"bfef2562-28c6-4640-a1c0-8a467bba8a79\" (UID: \"bfef2562-28c6-4640-a1c0-8a467bba8a79\") " Oct 30 20:03:07.145334 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.142527 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-data\") pod \"b913225f-a9cf-4af0-af6f-076a9c0ec683\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " Oct 30 20:03:07.145334 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.142711 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-data" pod "bfef2562-28c6-4640-a1c0-8a467bba8a79" (UID: "bfef2562-28c6-4640-a1c0-8a467bba8a79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:07.145334 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.142721 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-token\" (UniqueName: \"kubernetes.io/projected/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-token\") pod \"b913225f-a9cf-4af0-af6f-076a9c0ec683\" (UID: \"b913225f-a9cf-4af0-af6f-076a9c0ec683\") " Oct 30 20:03:07.145334 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.143090 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-workload-certs" pod "b913225f-a9cf-4af0-af6f-076a9c0ec683" (UID: "b913225f-a9cf-4af0-af6f-076a9c0ec683"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:07.145334 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.143627 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.145334 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.143653 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfef2562-28c6-4640-a1c0-8a467bba8a79-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.145334 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.143667 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.145334 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.143679 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.145334 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.145130 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-credential-socket" pod "b913225f-a9cf-4af0-af6f-076a9c0ec683" (UID: "b913225f-a9cf-4af0-af6f-076a9c0ec683"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:07.146230 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.145354 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-workload-certs" pod "bfef2562-28c6-4640-a1c0-8a467bba8a79" (UID: "bfef2562-28c6-4640-a1c0-8a467bba8a79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:07.146230 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.145704 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b913225f-a9cf-4af0-af6f-076a9c0ec683-istiod-ca-cert" pod "b913225f-a9cf-4af0-af6f-076a9c0ec683" (UID: "b913225f-a9cf-4af0-af6f-076a9c0ec683"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:07.146527 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.146501 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-envoy" pod "bfef2562-28c6-4640-a1c0-8a467bba8a79" (UID: "bfef2562-28c6-4640-a1c0-8a467bba8a79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:07.147158 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.146739 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-credential-socket" pod "bfef2562-28c6-4640-a1c0-8a467bba8a79" (UID: "bfef2562-28c6-4640-a1c0-8a467bba8a79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:07.147158 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.146857 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-token" pod "b913225f-a9cf-4af0-af6f-076a9c0ec683" (UID: "b913225f-a9cf-4af0-af6f-076a9c0ec683"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:07.147158 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.146891 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b913225f-a9cf-4af0-af6f-076a9c0ec683-kube-api-access-97rkd" pod "b913225f-a9cf-4af0-af6f-076a9c0ec683" (UID: "b913225f-a9cf-4af0-af6f-076a9c0ec683"). InnerVolumeSpecName "kube-api-access-97rkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:07.147158 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.147041 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-workload-socket" pod "bfef2562-28c6-4640-a1c0-8a467bba8a79" (UID: "bfef2562-28c6-4640-a1c0-8a467bba8a79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:07.147158 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.147105 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-token" pod "bfef2562-28c6-4640-a1c0-8a467bba8a79" (UID: "bfef2562-28c6-4640-a1c0-8a467bba8a79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:07.147158 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.147129 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-data" pod "b913225f-a9cf-4af0-af6f-076a9c0ec683" (UID: "b913225f-a9cf-4af0-af6f-076a9c0ec683"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:07.148037 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.147984 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-podinfo" pod "bfef2562-28c6-4640-a1c0-8a467bba8a79" (UID: "bfef2562-28c6-4640-a1c0-8a467bba8a79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:07.148213 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.148189 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-envoy" pod "b913225f-a9cf-4af0-af6f-076a9c0ec683" (UID: "b913225f-a9cf-4af0-af6f-076a9c0ec683"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:07.148367 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.148349 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-podinfo" pod "b913225f-a9cf-4af0-af6f-076a9c0ec683" (UID: "b913225f-a9cf-4af0-af6f-076a9c0ec683"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:07.148465 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.148386 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfef2562-28c6-4640-a1c0-8a467bba8a79-kube-api-access-hffxn" pod "bfef2562-28c6-4640-a1c0-8a467bba8a79" (UID: "bfef2562-28c6-4640-a1c0-8a467bba8a79"). InnerVolumeSpecName "kube-api-access-hffxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:07.246030 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.245921 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.247269 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.247246 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.247376 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.247365 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.247470 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.247459 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.247559 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.247544 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97rkd\" (UniqueName: \"kubernetes.io/projected/b913225f-a9cf-4af0-af6f-076a9c0ec683-kube-api-access-97rkd\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.247640 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.247628 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.247728 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.247716 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hffxn\" (UniqueName: \"kubernetes.io/projected/bfef2562-28c6-4640-a1c0-8a467bba8a79-kube-api-access-hffxn\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.247826 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.247804 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b913225f-a9cf-4af0-af6f-076a9c0ec683-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.247826 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.247826 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.248074 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.247841 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.248074 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.247855 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfef2562-28c6-4640-a1c0-8a467bba8a79-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.248074 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.247869 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef2562-28c6-4640-a1c0-8a467bba8a79-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.248074 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.247916 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:07.248074 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:07.247932 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b913225f-a9cf-4af0-af6f-076a9c0ec683-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:08.030755 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:08.030666 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt" Oct 30 20:03:08.032096 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:08.032063 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" event={"ID":"b913225f-a9cf-4af0-af6f-076a9c0ec683","Type":"ContainerDied","Data":"dda261ab9bf4cf378471d3239f65594bb09eb99d4089e3c7f46859e8808f983e"} Oct 30 20:03:08.032096 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:08.032087 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt" Oct 30 20:03:08.032332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:08.032166 5047 scope.go:122] "RemoveContainer" containerID="8b40dc18b077766fa4a849a56062cf6dc7ea6135eb0969a71834480c484d80c5" Oct 30 20:03:08.090234 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:08.090199 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt"] Oct 30 20:03:08.099848 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:08.099818 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-5thrt"] Oct 30 20:03:08.123337 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:08.123307 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt"] Oct 30 20:03:08.128760 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:08.128735 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9eb76-openshift-default-b7d55666c-j48zt"] Oct 30 20:03:09.048912 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:09.048877 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b913225f-a9cf-4af0-af6f-076a9c0ec683" path="/var/lib/kubelet/pods/b913225f-a9cf-4af0-af6f-076a9c0ec683/volumes" Oct 30 20:03:09.049461 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:09.049442 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfef2562-28c6-4640-a1c0-8a467bba8a79" path="/var/lib/kubelet/pods/bfef2562-28c6-4640-a1c0-8a467bba8a79/volumes" Oct 30 20:03:09.600960 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:03:09.600923 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:03:11.431795 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:11.431760 5047 scope.go:122] "RemoveContainer" containerID="da31b0f320d251bf4d850a8d683602d67c122471a95cea3e4c685b499381a20a" Oct 30 20:03:12.649295 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:12.649258 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr"] Oct 30 20:03:15.705786 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:15.705737 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr"] Oct 30 20:03:15.706318 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:15.706037 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" podUID="d4b63ca7-418b-44c0-a603-575c14f97efb" containerName="maas-api" containerID="cri-o://67a18d6f9c1680a5bdc11d17927a2c16cb514b48981ec131c6adc7cc0560e7de" gracePeriod=30 Oct 30 20:03:15.985410 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:15.985379 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 20:03:16.081660 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.081631 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4b63ca7-418b-44c0-a603-575c14f97efb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b63ca7-418b-44c0-a603-575c14f97efb-openshift-service-ca\") pod \"d4b63ca7-418b-44c0-a603-575c14f97efb\" (UID: \"d4b63ca7-418b-44c0-a603-575c14f97efb\") " Oct 30 20:03:16.081857 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.081753 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4b63ca7-418b-44c0-a603-575c14f97efb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b63ca7-418b-44c0-a603-575c14f97efb-metrics-certs\") pod \"d4b63ca7-418b-44c0-a603-575c14f97efb\" (UID: \"d4b63ca7-418b-44c0-a603-575c14f97efb\") " Oct 30 20:03:16.081857 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.081848 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4b63ca7-418b-44c0-a603-575c14f97efb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b63ca7-418b-44c0-a603-575c14f97efb-maas-api-tls\") pod \"d4b63ca7-418b-44c0-a603-575c14f97efb\" (UID: \"d4b63ca7-418b-44c0-a603-575c14f97efb\") " Oct 30 20:03:16.081985 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.081903 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b63ca7-418b-44c0-a603-575c14f97efb-kube-api-access-82bx6\" (UniqueName: \"kubernetes.io/projected/d4b63ca7-418b-44c0-a603-575c14f97efb-kube-api-access-82bx6\") pod \"d4b63ca7-418b-44c0-a603-575c14f97efb\" (UID: \"d4b63ca7-418b-44c0-a603-575c14f97efb\") " Oct 30 20:03:16.082092 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.082046 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b63ca7-418b-44c0-a603-575c14f97efb-openshift-service-ca" pod "d4b63ca7-418b-44c0-a603-575c14f97efb" (UID: "d4b63ca7-418b-44c0-a603-575c14f97efb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:16.082387 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.082364 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b63ca7-418b-44c0-a603-575c14f97efb-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:16.082963 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.082934 5047 generic.go:356] "Generic (PLEG): container finished" podID="d4b63ca7-418b-44c0-a603-575c14f97efb" containerID="67a18d6f9c1680a5bdc11d17927a2c16cb514b48981ec131c6adc7cc0560e7de" exitCode=0 Oct 30 20:03:16.083132 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.082972 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" Oct 30 20:03:16.083132 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.083029 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" event={"ID":"d4b63ca7-418b-44c0-a603-575c14f97efb","Type":"ContainerDied","Data":"67a18d6f9c1680a5bdc11d17927a2c16cb514b48981ec131c6adc7cc0560e7de"} Oct 30 20:03:16.083132 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.083088 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr" event={"ID":"d4b63ca7-418b-44c0-a603-575c14f97efb","Type":"ContainerDied","Data":"d87a4ccac66f5f77185b33b072405c8880c3ca3c7fdb2646816f1497deed404b"} Oct 30 20:03:16.083132 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.083117 5047 scope.go:122] "RemoveContainer" containerID="67a18d6f9c1680a5bdc11d17927a2c16cb514b48981ec131c6adc7cc0560e7de" Oct 30 20:03:16.084181 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.084155 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b63ca7-418b-44c0-a603-575c14f97efb-metrics-certs" pod "d4b63ca7-418b-44c0-a603-575c14f97efb" (UID: "d4b63ca7-418b-44c0-a603-575c14f97efb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:16.084304 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.084282 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b63ca7-418b-44c0-a603-575c14f97efb-kube-api-access-82bx6" pod "d4b63ca7-418b-44c0-a603-575c14f97efb" (UID: "d4b63ca7-418b-44c0-a603-575c14f97efb"). InnerVolumeSpecName "kube-api-access-82bx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:16.084371 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.084310 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b63ca7-418b-44c0-a603-575c14f97efb-maas-api-tls" pod "d4b63ca7-418b-44c0-a603-575c14f97efb" (UID: "d4b63ca7-418b-44c0-a603-575c14f97efb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:16.109699 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.109677 5047 scope.go:122] "RemoveContainer" containerID="67a18d6f9c1680a5bdc11d17927a2c16cb514b48981ec131c6adc7cc0560e7de" Oct 30 20:03:16.110067 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:03:16.110040 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a18d6f9c1680a5bdc11d17927a2c16cb514b48981ec131c6adc7cc0560e7de\": container with ID starting with 67a18d6f9c1680a5bdc11d17927a2c16cb514b48981ec131c6adc7cc0560e7de not found: ID does not exist" containerID="67a18d6f9c1680a5bdc11d17927a2c16cb514b48981ec131c6adc7cc0560e7de" Oct 30 20:03:16.110157 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.110086 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a18d6f9c1680a5bdc11d17927a2c16cb514b48981ec131c6adc7cc0560e7de"} err="failed to get container status \"67a18d6f9c1680a5bdc11d17927a2c16cb514b48981ec131c6adc7cc0560e7de\": rpc error: code = NotFound desc = could not find container \"67a18d6f9c1680a5bdc11d17927a2c16cb514b48981ec131c6adc7cc0560e7de\": container with ID starting with 67a18d6f9c1680a5bdc11d17927a2c16cb514b48981ec131c6adc7cc0560e7de not found: ID does not exist" Oct 30 20:03:16.184041 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.183971 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b63ca7-418b-44c0-a603-575c14f97efb-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:16.184272 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.184252 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b63ca7-418b-44c0-a603-575c14f97efb-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:16.184371 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.184277 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82bx6\" (UniqueName: \"kubernetes.io/projected/d4b63ca7-418b-44c0-a603-575c14f97efb-kube-api-access-82bx6\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:16.425069 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.425031 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr"] Oct 30 20:03:16.435350 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:16.435323 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-095f55-566f66b8c9-g7fgr"] Oct 30 20:03:17.050066 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:17.049998 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b63ca7-418b-44c0-a603-575c14f97efb" path="/var/lib/kubelet/pods/d4b63ca7-418b-44c0-a603-575c14f97efb/volumes" Oct 30 20:03:17.649814 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:17.649750 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" podUID="71ff13b0-8eb3-474b-8579-29b192ccd9ee" containerName="payload-pre-processing" containerID="cri-o://f7934e0fb31562f7d0f762abdc8c3ebb373602d38c9473c132056d0e9ea71180" gracePeriod=25 Oct 30 20:03:22.815699 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:22.815661 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" podUID="71ff13b0-8eb3-474b-8579-29b192ccd9ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:29.566041 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:29.565991 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b"] Oct 30 20:03:29.567223 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:29.566284 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" podUID="3d40dfe2-d6ef-449d-a678-8e6938ea132e" containerName="maas-api" containerID="cri-o://39d8f08885c12974d504ebe9383d76d73d8363fdec4e0442b6c9df80f6bf1a25" gracePeriod=30 Oct 30 20:03:29.846334 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:29.846302 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 20:03:29.935417 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:29.935372 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d40dfe2-d6ef-449d-a678-8e6938ea132e-kube-api-access-rlsr5\" (UniqueName: \"kubernetes.io/projected/3d40dfe2-d6ef-449d-a678-8e6938ea132e-kube-api-access-rlsr5\") pod \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\" (UID: \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\") " Oct 30 20:03:29.935620 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:29.935435 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d40dfe2-d6ef-449d-a678-8e6938ea132e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d40dfe2-d6ef-449d-a678-8e6938ea132e-maas-api-tls\") pod \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\" (UID: \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\") " Oct 30 20:03:29.935620 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:29.935530 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d40dfe2-d6ef-449d-a678-8e6938ea132e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d40dfe2-d6ef-449d-a678-8e6938ea132e-openshift-service-ca\") pod \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\" (UID: \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\") " Oct 30 20:03:29.935756 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:29.935645 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d40dfe2-d6ef-449d-a678-8e6938ea132e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d40dfe2-d6ef-449d-a678-8e6938ea132e-metrics-certs\") pod \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\" (UID: \"3d40dfe2-d6ef-449d-a678-8e6938ea132e\") " Oct 30 20:03:29.935948 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:29.935918 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d40dfe2-d6ef-449d-a678-8e6938ea132e-openshift-service-ca" pod "3d40dfe2-d6ef-449d-a678-8e6938ea132e" (UID: "3d40dfe2-d6ef-449d-a678-8e6938ea132e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:29.936256 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:29.936230 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d40dfe2-d6ef-449d-a678-8e6938ea132e-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:29.937656 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:29.937625 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d40dfe2-d6ef-449d-a678-8e6938ea132e-maas-api-tls" pod "3d40dfe2-d6ef-449d-a678-8e6938ea132e" (UID: "3d40dfe2-d6ef-449d-a678-8e6938ea132e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:29.938073 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:29.938052 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d40dfe2-d6ef-449d-a678-8e6938ea132e-kube-api-access-rlsr5" pod "3d40dfe2-d6ef-449d-a678-8e6938ea132e" (UID: "3d40dfe2-d6ef-449d-a678-8e6938ea132e"). InnerVolumeSpecName "kube-api-access-rlsr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:29.938184 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:29.938160 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d40dfe2-d6ef-449d-a678-8e6938ea132e-metrics-certs" pod "3d40dfe2-d6ef-449d-a678-8e6938ea132e" (UID: "3d40dfe2-d6ef-449d-a678-8e6938ea132e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:30.037144 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:30.037107 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlsr5\" (UniqueName: \"kubernetes.io/projected/3d40dfe2-d6ef-449d-a678-8e6938ea132e-kube-api-access-rlsr5\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:30.037144 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:30.037142 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d40dfe2-d6ef-449d-a678-8e6938ea132e-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:30.037144 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:30.037155 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d40dfe2-d6ef-449d-a678-8e6938ea132e-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:30.164340 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:30.164297 5047 generic.go:356] "Generic (PLEG): container finished" podID="3d40dfe2-d6ef-449d-a678-8e6938ea132e" containerID="39d8f08885c12974d504ebe9383d76d73d8363fdec4e0442b6c9df80f6bf1a25" exitCode=0 Oct 30 20:03:30.164542 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:30.164362 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" Oct 30 20:03:30.164542 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:30.164378 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" event={"ID":"3d40dfe2-d6ef-449d-a678-8e6938ea132e","Type":"ContainerDied","Data":"39d8f08885c12974d504ebe9383d76d73d8363fdec4e0442b6c9df80f6bf1a25"} Oct 30 20:03:30.164542 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:30.164465 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b" event={"ID":"3d40dfe2-d6ef-449d-a678-8e6938ea132e","Type":"ContainerDied","Data":"bfb0e4858b73b9583184fb00379a44d03c98b9c1df87f296756ec40721ff1f00"} Oct 30 20:03:30.164542 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:30.164492 5047 scope.go:122] "RemoveContainer" containerID="39d8f08885c12974d504ebe9383d76d73d8363fdec4e0442b6c9df80f6bf1a25" Oct 30 20:03:30.175502 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:30.175478 5047 scope.go:122] "RemoveContainer" containerID="39d8f08885c12974d504ebe9383d76d73d8363fdec4e0442b6c9df80f6bf1a25" Oct 30 20:03:30.175829 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:03:30.175807 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d8f08885c12974d504ebe9383d76d73d8363fdec4e0442b6c9df80f6bf1a25\": container with ID starting with 39d8f08885c12974d504ebe9383d76d73d8363fdec4e0442b6c9df80f6bf1a25 not found: ID does not exist" containerID="39d8f08885c12974d504ebe9383d76d73d8363fdec4e0442b6c9df80f6bf1a25" Oct 30 20:03:30.175886 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:30.175839 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d8f08885c12974d504ebe9383d76d73d8363fdec4e0442b6c9df80f6bf1a25"} err="failed to get container status \"39d8f08885c12974d504ebe9383d76d73d8363fdec4e0442b6c9df80f6bf1a25\": rpc error: code = NotFound desc = could not find container \"39d8f08885c12974d504ebe9383d76d73d8363fdec4e0442b6c9df80f6bf1a25\": container with ID starting with 39d8f08885c12974d504ebe9383d76d73d8363fdec4e0442b6c9df80f6bf1a25 not found: ID does not exist" Oct 30 20:03:30.208527 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:30.208480 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b"] Oct 30 20:03:30.214042 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:30.213991 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-ec2fe7-7bfc58cf96-g9h8b"] Oct 30 20:03:31.049544 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:31.049508 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d40dfe2-d6ef-449d-a678-8e6938ea132e" path="/var/lib/kubelet/pods/3d40dfe2-d6ef-449d-a678-8e6938ea132e/volumes" Oct 30 20:03:32.814741 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:32.814695 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" podUID="71ff13b0-8eb3-474b-8579-29b192ccd9ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:03:33.663344 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:33.663309 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp"] Oct 30 20:03:33.663597 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:33.663570 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" podUID="ea906da9-9a7c-4755-b5f1-2f6388875be2" containerName="istio-proxy" containerID="cri-o://2c6a99184a1d0a2bc0227341117cc83a4c2084c1dbe547a6ae90ee730309920e" gracePeriod=30 Oct 30 20:03:35.247146 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:35.247108 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995"] Oct 30 20:03:35.248025 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:35.247972 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" podUID="a12a3bdc-6b65-4496-8458-ff694a332732" containerName="maas-api" containerID="cri-o://b303640fedc1055469b55afce06b14c662a1a1028903a5df44fab6d3c6abcc91" gracePeriod=30 Oct 30 20:03:35.590571 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:35.590515 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 20:03:35.720778 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:35.720732 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a12a3bdc-6b65-4496-8458-ff694a332732-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12a3bdc-6b65-4496-8458-ff694a332732-metrics-certs\") pod \"a12a3bdc-6b65-4496-8458-ff694a332732\" (UID: \"a12a3bdc-6b65-4496-8458-ff694a332732\") " Oct 30 20:03:35.720997 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:35.720886 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a12a3bdc-6b65-4496-8458-ff694a332732-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12a3bdc-6b65-4496-8458-ff694a332732-openshift-service-ca\") pod \"a12a3bdc-6b65-4496-8458-ff694a332732\" (UID: \"a12a3bdc-6b65-4496-8458-ff694a332732\") " Oct 30 20:03:35.720997 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:35.720926 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a12a3bdc-6b65-4496-8458-ff694a332732-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12a3bdc-6b65-4496-8458-ff694a332732-maas-api-tls\") pod \"a12a3bdc-6b65-4496-8458-ff694a332732\" (UID: \"a12a3bdc-6b65-4496-8458-ff694a332732\") " Oct 30 20:03:35.720997 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:35.720955 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a12a3bdc-6b65-4496-8458-ff694a332732-kube-api-access-vm6bc\" (UniqueName: \"kubernetes.io/projected/a12a3bdc-6b65-4496-8458-ff694a332732-kube-api-access-vm6bc\") pod \"a12a3bdc-6b65-4496-8458-ff694a332732\" (UID: \"a12a3bdc-6b65-4496-8458-ff694a332732\") " Oct 30 20:03:35.721355 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:35.721325 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12a3bdc-6b65-4496-8458-ff694a332732-openshift-service-ca" pod "a12a3bdc-6b65-4496-8458-ff694a332732" (UID: "a12a3bdc-6b65-4496-8458-ff694a332732"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:35.722856 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:35.722833 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a12a3bdc-6b65-4496-8458-ff694a332732-metrics-certs" pod "a12a3bdc-6b65-4496-8458-ff694a332732" (UID: "a12a3bdc-6b65-4496-8458-ff694a332732"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:35.722939 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:35.722922 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a12a3bdc-6b65-4496-8458-ff694a332732-maas-api-tls" pod "a12a3bdc-6b65-4496-8458-ff694a332732" (UID: "a12a3bdc-6b65-4496-8458-ff694a332732"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:35.723116 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:35.723095 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12a3bdc-6b65-4496-8458-ff694a332732-kube-api-access-vm6bc" pod "a12a3bdc-6b65-4496-8458-ff694a332732" (UID: "a12a3bdc-6b65-4496-8458-ff694a332732"). InnerVolumeSpecName "kube-api-access-vm6bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:35.822698 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:35.822619 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12a3bdc-6b65-4496-8458-ff694a332732-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:35.822698 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:35.822649 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12a3bdc-6b65-4496-8458-ff694a332732-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:35.822698 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:35.822658 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12a3bdc-6b65-4496-8458-ff694a332732-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:35.822698 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:35.822667 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm6bc\" (UniqueName: \"kubernetes.io/projected/a12a3bdc-6b65-4496-8458-ff694a332732-kube-api-access-vm6bc\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:36.201124 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:36.201089 5047 generic.go:356] "Generic (PLEG): container finished" podID="a12a3bdc-6b65-4496-8458-ff694a332732" containerID="b303640fedc1055469b55afce06b14c662a1a1028903a5df44fab6d3c6abcc91" exitCode=0 Oct 30 20:03:36.201313 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:36.201142 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" Oct 30 20:03:36.201313 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:36.201155 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" event={"ID":"a12a3bdc-6b65-4496-8458-ff694a332732","Type":"ContainerDied","Data":"b303640fedc1055469b55afce06b14c662a1a1028903a5df44fab6d3c6abcc91"} Oct 30 20:03:36.201313 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:36.201213 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995" event={"ID":"a12a3bdc-6b65-4496-8458-ff694a332732","Type":"ContainerDied","Data":"3345e2304d7a1c43358a14c84dad6a79f1c83343aceacebd715c8b59ee3f7f4c"} Oct 30 20:03:36.201313 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:36.201236 5047 scope.go:122] "RemoveContainer" containerID="b303640fedc1055469b55afce06b14c662a1a1028903a5df44fab6d3c6abcc91" Oct 30 20:03:36.212087 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:36.212066 5047 scope.go:122] "RemoveContainer" containerID="b303640fedc1055469b55afce06b14c662a1a1028903a5df44fab6d3c6abcc91" Oct 30 20:03:36.212390 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:03:36.212372 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b303640fedc1055469b55afce06b14c662a1a1028903a5df44fab6d3c6abcc91\": container with ID starting with b303640fedc1055469b55afce06b14c662a1a1028903a5df44fab6d3c6abcc91 not found: ID does not exist" containerID="b303640fedc1055469b55afce06b14c662a1a1028903a5df44fab6d3c6abcc91" Oct 30 20:03:36.212428 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:36.212399 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b303640fedc1055469b55afce06b14c662a1a1028903a5df44fab6d3c6abcc91"} err="failed to get container status \"b303640fedc1055469b55afce06b14c662a1a1028903a5df44fab6d3c6abcc91\": rpc error: code = NotFound desc = could not find container \"b303640fedc1055469b55afce06b14c662a1a1028903a5df44fab6d3c6abcc91\": container with ID starting with b303640fedc1055469b55afce06b14c662a1a1028903a5df44fab6d3c6abcc91 not found: ID does not exist" Oct 30 20:03:36.248136 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:36.247918 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995"] Oct 30 20:03:36.256470 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:36.256438 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9efc72-59d769f774-6s995"] Oct 30 20:03:37.049986 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:37.049951 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a12a3bdc-6b65-4496-8458-ff694a332732" path="/var/lib/kubelet/pods/a12a3bdc-6b65-4496-8458-ff694a332732/volumes" Oct 30 20:03:37.796186 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:37.796147 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r"] Oct 30 20:03:37.796682 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:37.796412 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" podUID="ebafb097-ceaa-43ca-9ce4-023cdca5ad56" containerName="istio-proxy" containerID="cri-o://de9c6eecd09c5a1a215b0b83256539dea6b4c3f883eca61bad9dd2c87acefabe" gracePeriod=30 Oct 30 20:03:37.955320 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:37.955294 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" Oct 30 20:03:38.045508 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:38.045472 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ff13b0-8eb3-474b-8579-29b192ccd9ee-kube-api-access-88r8b\" (UniqueName: \"kubernetes.io/projected/71ff13b0-8eb3-474b-8579-29b192ccd9ee-kube-api-access-88r8b\") pod \"71ff13b0-8eb3-474b-8579-29b192ccd9ee\" (UID: \"71ff13b0-8eb3-474b-8579-29b192ccd9ee\") " Oct 30 20:03:38.045715 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:38.045535 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ff13b0-8eb3-474b-8579-29b192ccd9ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ff13b0-8eb3-474b-8579-29b192ccd9ee-plugins-config-volume\") pod \"71ff13b0-8eb3-474b-8579-29b192ccd9ee\" (UID: \"71ff13b0-8eb3-474b-8579-29b192ccd9ee\") " Oct 30 20:03:38.045935 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:38.045910 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ff13b0-8eb3-474b-8579-29b192ccd9ee-plugins-config-volume" pod "71ff13b0-8eb3-474b-8579-29b192ccd9ee" (UID: "71ff13b0-8eb3-474b-8579-29b192ccd9ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:38.047887 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:38.047859 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ff13b0-8eb3-474b-8579-29b192ccd9ee-kube-api-access-88r8b" pod "71ff13b0-8eb3-474b-8579-29b192ccd9ee" (UID: "71ff13b0-8eb3-474b-8579-29b192ccd9ee"). InnerVolumeSpecName "kube-api-access-88r8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:38.146983 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:38.146942 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88r8b\" (UniqueName: \"kubernetes.io/projected/71ff13b0-8eb3-474b-8579-29b192ccd9ee-kube-api-access-88r8b\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:38.146983 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:38.146981 5047 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ff13b0-8eb3-474b-8579-29b192ccd9ee-plugins-config-volume\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:38.214646 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:38.214562 5047 generic.go:356] "Generic (PLEG): container finished" podID="71ff13b0-8eb3-474b-8579-29b192ccd9ee" containerID="f7934e0fb31562f7d0f762abdc8c3ebb373602d38c9473c132056d0e9ea71180" exitCode=0 Oct 30 20:03:38.214646 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:38.214610 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" Oct 30 20:03:38.214830 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:38.214643 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" event={"ID":"71ff13b0-8eb3-474b-8579-29b192ccd9ee","Type":"ContainerDied","Data":"f7934e0fb31562f7d0f762abdc8c3ebb373602d38c9473c132056d0e9ea71180"} Oct 30 20:03:38.214830 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:38.214699 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr" event={"ID":"71ff13b0-8eb3-474b-8579-29b192ccd9ee","Type":"ContainerDied","Data":"97e4a938b07d321927645efc4bfdc110ff7c7f9ae0761670130530f2ce5a0c57"} Oct 30 20:03:38.214830 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:38.214720 5047 scope.go:122] "RemoveContainer" containerID="f7934e0fb31562f7d0f762abdc8c3ebb373602d38c9473c132056d0e9ea71180" Oct 30 20:03:38.224128 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:38.224111 5047 scope.go:122] "RemoveContainer" containerID="f7934e0fb31562f7d0f762abdc8c3ebb373602d38c9473c132056d0e9ea71180" Oct 30 20:03:38.224387 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:03:38.224369 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7934e0fb31562f7d0f762abdc8c3ebb373602d38c9473c132056d0e9ea71180\": container with ID starting with f7934e0fb31562f7d0f762abdc8c3ebb373602d38c9473c132056d0e9ea71180 not found: ID does not exist" containerID="f7934e0fb31562f7d0f762abdc8c3ebb373602d38c9473c132056d0e9ea71180" Oct 30 20:03:38.224448 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:38.224396 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7934e0fb31562f7d0f762abdc8c3ebb373602d38c9473c132056d0e9ea71180"} err="failed to get container status \"f7934e0fb31562f7d0f762abdc8c3ebb373602d38c9473c132056d0e9ea71180\": rpc error: code = NotFound desc = could not find container \"f7934e0fb31562f7d0f762abdc8c3ebb373602d38c9473c132056d0e9ea71180\": container with ID starting with f7934e0fb31562f7d0f762abdc8c3ebb373602d38c9473c132056d0e9ea71180 not found: ID does not exist" Oct 30 20:03:38.259298 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:38.259272 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr"] Oct 30 20:03:38.262875 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:38.262853 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9efc72-899979b9f-96hcr"] Oct 30 20:03:38.925447 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:38.925421 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 20:03:39.048542 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.048458 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ff13b0-8eb3-474b-8579-29b192ccd9ee" path="/var/lib/kubelet/pods/71ff13b0-8eb3-474b-8579-29b192ccd9ee/volumes" Oct 30 20:03:39.056780 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.056749 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-token\" (UniqueName: \"kubernetes.io/projected/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-token\") pod \"ea906da9-9a7c-4755-b5f1-2f6388875be2\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " Oct 30 20:03:39.056936 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.056803 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-envoy\") pod \"ea906da9-9a7c-4755-b5f1-2f6388875be2\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " Oct 30 20:03:39.056936 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.056843 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea906da9-9a7c-4755-b5f1-2f6388875be2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea906da9-9a7c-4755-b5f1-2f6388875be2-istiod-ca-cert\") pod \"ea906da9-9a7c-4755-b5f1-2f6388875be2\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " Oct 30 20:03:39.056936 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.056914 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-data\") pod \"ea906da9-9a7c-4755-b5f1-2f6388875be2\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " Oct 30 20:03:39.057150 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.057023 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-workload-certs\") pod \"ea906da9-9a7c-4755-b5f1-2f6388875be2\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " Oct 30 20:03:39.057150 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.057054 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-workload-socket\") pod \"ea906da9-9a7c-4755-b5f1-2f6388875be2\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " Oct 30 20:03:39.057150 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.057089 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-podinfo\") pod \"ea906da9-9a7c-4755-b5f1-2f6388875be2\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " Oct 30 20:03:39.057321 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.057147 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea906da9-9a7c-4755-b5f1-2f6388875be2-kube-api-access-d65tb\" (UniqueName: \"kubernetes.io/projected/ea906da9-9a7c-4755-b5f1-2f6388875be2-kube-api-access-d65tb\") pod \"ea906da9-9a7c-4755-b5f1-2f6388875be2\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " Oct 30 20:03:39.057321 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.057198 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-credential-socket\") pod \"ea906da9-9a7c-4755-b5f1-2f6388875be2\" (UID: \"ea906da9-9a7c-4755-b5f1-2f6388875be2\") " Oct 30 20:03:39.057321 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.057262 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-data" pod "ea906da9-9a7c-4755-b5f1-2f6388875be2" (UID: "ea906da9-9a7c-4755-b5f1-2f6388875be2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:39.057499 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.057342 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-workload-socket" pod "ea906da9-9a7c-4755-b5f1-2f6388875be2" (UID: "ea906da9-9a7c-4755-b5f1-2f6388875be2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:39.057563 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.057524 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-credential-socket" pod "ea906da9-9a7c-4755-b5f1-2f6388875be2" (UID: "ea906da9-9a7c-4755-b5f1-2f6388875be2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:39.057662 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.057645 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:39.057736 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.057690 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:39.057736 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.057710 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:39.057844 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.057646 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-workload-certs" pod "ea906da9-9a7c-4755-b5f1-2f6388875be2" (UID: "ea906da9-9a7c-4755-b5f1-2f6388875be2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:39.057844 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.057708 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea906da9-9a7c-4755-b5f1-2f6388875be2-istiod-ca-cert" pod "ea906da9-9a7c-4755-b5f1-2f6388875be2" (UID: "ea906da9-9a7c-4755-b5f1-2f6388875be2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:39.059457 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.059431 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-envoy" pod "ea906da9-9a7c-4755-b5f1-2f6388875be2" (UID: "ea906da9-9a7c-4755-b5f1-2f6388875be2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:39.059583 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.059476 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-token" pod "ea906da9-9a7c-4755-b5f1-2f6388875be2" (UID: "ea906da9-9a7c-4755-b5f1-2f6388875be2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:39.059917 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.059891 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-podinfo" pod "ea906da9-9a7c-4755-b5f1-2f6388875be2" (UID: "ea906da9-9a7c-4755-b5f1-2f6388875be2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:39.059917 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.059905 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea906da9-9a7c-4755-b5f1-2f6388875be2-kube-api-access-d65tb" pod "ea906da9-9a7c-4755-b5f1-2f6388875be2" (UID: "ea906da9-9a7c-4755-b5f1-2f6388875be2"). InnerVolumeSpecName "kube-api-access-d65tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:39.158855 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.158821 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:39.158855 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.158851 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:39.158855 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.158861 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d65tb\" (UniqueName: \"kubernetes.io/projected/ea906da9-9a7c-4755-b5f1-2f6388875be2-kube-api-access-d65tb\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:39.159135 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.158872 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:39.159135 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.158884 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea906da9-9a7c-4755-b5f1-2f6388875be2-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:39.159135 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.158897 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea906da9-9a7c-4755-b5f1-2f6388875be2-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:39.222380 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.222345 5047 generic.go:356] "Generic (PLEG): container finished" podID="ea906da9-9a7c-4755-b5f1-2f6388875be2" containerID="2c6a99184a1d0a2bc0227341117cc83a4c2084c1dbe547a6ae90ee730309920e" exitCode=0 Oct 30 20:03:39.222551 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.222403 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" Oct 30 20:03:39.222551 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.222424 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" event={"ID":"ea906da9-9a7c-4755-b5f1-2f6388875be2","Type":"ContainerDied","Data":"2c6a99184a1d0a2bc0227341117cc83a4c2084c1dbe547a6ae90ee730309920e"} Oct 30 20:03:39.222551 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.222489 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp" event={"ID":"ea906da9-9a7c-4755-b5f1-2f6388875be2","Type":"ContainerDied","Data":"ef935aadaeedd8307417d686138367607c6c5d013bf4e6e87d2660b020c8f052"} Oct 30 20:03:39.222551 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.222514 5047 scope.go:122] "RemoveContainer" containerID="2c6a99184a1d0a2bc0227341117cc83a4c2084c1dbe547a6ae90ee730309920e" Oct 30 20:03:39.232700 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.232682 5047 scope.go:122] "RemoveContainer" containerID="004460907acd6a0cfa6d797c7623ed7e37abdd83f0532b4b4862c59d34dbd984" Oct 30 20:03:39.244276 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.244254 5047 scope.go:122] "RemoveContainer" containerID="2c6a99184a1d0a2bc0227341117cc83a4c2084c1dbe547a6ae90ee730309920e" Oct 30 20:03:39.244635 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:03:39.244612 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6a99184a1d0a2bc0227341117cc83a4c2084c1dbe547a6ae90ee730309920e\": container with ID starting with 2c6a99184a1d0a2bc0227341117cc83a4c2084c1dbe547a6ae90ee730309920e not found: ID does not exist" containerID="2c6a99184a1d0a2bc0227341117cc83a4c2084c1dbe547a6ae90ee730309920e" Oct 30 20:03:39.244711 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.244647 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6a99184a1d0a2bc0227341117cc83a4c2084c1dbe547a6ae90ee730309920e"} err="failed to get container status \"2c6a99184a1d0a2bc0227341117cc83a4c2084c1dbe547a6ae90ee730309920e\": rpc error: code = NotFound desc = could not find container \"2c6a99184a1d0a2bc0227341117cc83a4c2084c1dbe547a6ae90ee730309920e\": container with ID starting with 2c6a99184a1d0a2bc0227341117cc83a4c2084c1dbe547a6ae90ee730309920e not found: ID does not exist" Oct 30 20:03:39.244711 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.244671 5047 scope.go:122] "RemoveContainer" containerID="004460907acd6a0cfa6d797c7623ed7e37abdd83f0532b4b4862c59d34dbd984" Oct 30 20:03:39.244926 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:03:39.244911 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004460907acd6a0cfa6d797c7623ed7e37abdd83f0532b4b4862c59d34dbd984\": container with ID starting with 004460907acd6a0cfa6d797c7623ed7e37abdd83f0532b4b4862c59d34dbd984 not found: ID does not exist" containerID="004460907acd6a0cfa6d797c7623ed7e37abdd83f0532b4b4862c59d34dbd984" Oct 30 20:03:39.244966 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.244930 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004460907acd6a0cfa6d797c7623ed7e37abdd83f0532b4b4862c59d34dbd984"} err="failed to get container status \"004460907acd6a0cfa6d797c7623ed7e37abdd83f0532b4b4862c59d34dbd984\": rpc error: code = NotFound desc = could not find container \"004460907acd6a0cfa6d797c7623ed7e37abdd83f0532b4b4862c59d34dbd984\": container with ID starting with 004460907acd6a0cfa6d797c7623ed7e37abdd83f0532b4b4862c59d34dbd984 not found: ID does not exist" Oct 30 20:03:39.270829 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.270796 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp"] Oct 30 20:03:39.280840 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:39.280811 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-095f55-openshift-default-6556fb64bc-xfhsp"] Oct 30 20:03:39.601670 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:03:39.601634 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:03:41.051145 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:41.051112 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea906da9-9a7c-4755-b5f1-2f6388875be2" path="/var/lib/kubelet/pods/ea906da9-9a7c-4755-b5f1-2f6388875be2/volumes" Oct 30 20:03:41.465801 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:41.465762 5047 patch_prober.go:29] interesting pod/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:03:41.465985 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:41.465826 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" podUID="ebafb097-ceaa-43ca-9ce4-023cdca5ad56" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:03:43.076740 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.076671 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 20:03:43.096423 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.096388 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-credential-socket\") pod \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " Oct 30 20:03:43.096578 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.096443 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istiod-ca-cert\") pod \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " Oct 30 20:03:43.096578 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.096486 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-workload-certs\") pod \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " Oct 30 20:03:43.096578 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.096526 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-workload-socket\") pod \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " Oct 30 20:03:43.096719 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.096637 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-envoy\") pod \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " Oct 30 20:03:43.096719 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.096643 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-credential-socket" pod "ebafb097-ceaa-43ca-9ce4-023cdca5ad56" (UID: "ebafb097-ceaa-43ca-9ce4-023cdca5ad56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:43.096719 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.096677 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-kube-api-access-wxf2x\" (UniqueName: \"kubernetes.io/projected/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-kube-api-access-wxf2x\") pod \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " Oct 30 20:03:43.096719 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.096708 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-podinfo\") pod \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " Oct 30 20:03:43.096869 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.096768 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-token\" (UniqueName: \"kubernetes.io/projected/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-token\") pod \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " Oct 30 20:03:43.096869 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.096812 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-data\") pod \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\" (UID: \"ebafb097-ceaa-43ca-9ce4-023cdca5ad56\") " Oct 30 20:03:43.096869 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.096794 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-workload-certs" pod "ebafb097-ceaa-43ca-9ce4-023cdca5ad56" (UID: "ebafb097-ceaa-43ca-9ce4-023cdca5ad56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:43.096990 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.096875 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-workload-socket" pod "ebafb097-ceaa-43ca-9ce4-023cdca5ad56" (UID: "ebafb097-ceaa-43ca-9ce4-023cdca5ad56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:43.096990 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.096884 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istiod-ca-cert" pod "ebafb097-ceaa-43ca-9ce4-023cdca5ad56" (UID: "ebafb097-ceaa-43ca-9ce4-023cdca5ad56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:43.097712 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.097252 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:43.097712 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.097275 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:43.097712 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.097289 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:43.097712 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.097305 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:43.097712 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.097347 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-data" pod "ebafb097-ceaa-43ca-9ce4-023cdca5ad56" (UID: "ebafb097-ceaa-43ca-9ce4-023cdca5ad56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:43.099001 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.098973 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-podinfo" pod "ebafb097-ceaa-43ca-9ce4-023cdca5ad56" (UID: "ebafb097-ceaa-43ca-9ce4-023cdca5ad56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:43.099126 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.099104 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-kube-api-access-wxf2x" pod "ebafb097-ceaa-43ca-9ce4-023cdca5ad56" (UID: "ebafb097-ceaa-43ca-9ce4-023cdca5ad56"). InnerVolumeSpecName "kube-api-access-wxf2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:43.099260 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.099239 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-token" pod "ebafb097-ceaa-43ca-9ce4-023cdca5ad56" (UID: "ebafb097-ceaa-43ca-9ce4-023cdca5ad56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:43.099363 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.099256 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-envoy" pod "ebafb097-ceaa-43ca-9ce4-023cdca5ad56" (UID: "ebafb097-ceaa-43ca-9ce4-023cdca5ad56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:43.197968 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.197930 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:43.197968 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.197961 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:43.197968 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.197970 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:43.198236 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.197980 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxf2x\" (UniqueName: \"kubernetes.io/projected/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-kube-api-access-wxf2x\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:43.198236 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.197989 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebafb097-ceaa-43ca-9ce4-023cdca5ad56-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:43.250284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.250244 5047 generic.go:356] "Generic (PLEG): container finished" podID="ebafb097-ceaa-43ca-9ce4-023cdca5ad56" containerID="de9c6eecd09c5a1a215b0b83256539dea6b4c3f883eca61bad9dd2c87acefabe" exitCode=0 Oct 30 20:03:43.250458 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.250295 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" event={"ID":"ebafb097-ceaa-43ca-9ce4-023cdca5ad56","Type":"ContainerDied","Data":"de9c6eecd09c5a1a215b0b83256539dea6b4c3f883eca61bad9dd2c87acefabe"} Oct 30 20:03:43.250458 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.250336 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" Oct 30 20:03:43.250458 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.250366 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r" event={"ID":"ebafb097-ceaa-43ca-9ce4-023cdca5ad56","Type":"ContainerDied","Data":"5f2638ce77a4e2bc50a52414638618bf6a6ccdf04aadbc7a4605b370ee9ee39b"} Oct 30 20:03:43.250458 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.250398 5047 scope.go:122] "RemoveContainer" containerID="de9c6eecd09c5a1a215b0b83256539dea6b4c3f883eca61bad9dd2c87acefabe" Oct 30 20:03:43.260857 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.260830 5047 scope.go:122] "RemoveContainer" containerID="953e9794a673ece6ad8731c54373acb7f6362680a74e0be35a4a680dd8b1cdc7" Oct 30 20:03:43.269666 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.269640 5047 scope.go:122] "RemoveContainer" containerID="de9c6eecd09c5a1a215b0b83256539dea6b4c3f883eca61bad9dd2c87acefabe" Oct 30 20:03:43.269971 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:03:43.269949 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de9c6eecd09c5a1a215b0b83256539dea6b4c3f883eca61bad9dd2c87acefabe\": container with ID starting with de9c6eecd09c5a1a215b0b83256539dea6b4c3f883eca61bad9dd2c87acefabe not found: ID does not exist" containerID="de9c6eecd09c5a1a215b0b83256539dea6b4c3f883eca61bad9dd2c87acefabe" Oct 30 20:03:43.270092 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.269978 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de9c6eecd09c5a1a215b0b83256539dea6b4c3f883eca61bad9dd2c87acefabe"} err="failed to get container status \"de9c6eecd09c5a1a215b0b83256539dea6b4c3f883eca61bad9dd2c87acefabe\": rpc error: code = NotFound desc = could not find container \"de9c6eecd09c5a1a215b0b83256539dea6b4c3f883eca61bad9dd2c87acefabe\": container with ID starting with de9c6eecd09c5a1a215b0b83256539dea6b4c3f883eca61bad9dd2c87acefabe not found: ID does not exist" Oct 30 20:03:43.270092 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.269998 5047 scope.go:122] "RemoveContainer" containerID="953e9794a673ece6ad8731c54373acb7f6362680a74e0be35a4a680dd8b1cdc7" Oct 30 20:03:43.270313 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:03:43.270296 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"953e9794a673ece6ad8731c54373acb7f6362680a74e0be35a4a680dd8b1cdc7\": container with ID starting with 953e9794a673ece6ad8731c54373acb7f6362680a74e0be35a4a680dd8b1cdc7 not found: ID does not exist" containerID="953e9794a673ece6ad8731c54373acb7f6362680a74e0be35a4a680dd8b1cdc7" Oct 30 20:03:43.270360 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.270317 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"953e9794a673ece6ad8731c54373acb7f6362680a74e0be35a4a680dd8b1cdc7"} err="failed to get container status \"953e9794a673ece6ad8731c54373acb7f6362680a74e0be35a4a680dd8b1cdc7\": rpc error: code = NotFound desc = could not find container \"953e9794a673ece6ad8731c54373acb7f6362680a74e0be35a4a680dd8b1cdc7\": container with ID starting with 953e9794a673ece6ad8731c54373acb7f6362680a74e0be35a4a680dd8b1cdc7 not found: ID does not exist" Oct 30 20:03:43.295218 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.295183 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r"] Oct 30 20:03:43.301846 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:43.301817 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-ec2fe7-openshift-default-85c54cbbb5-cbv7r"] Oct 30 20:03:45.048695 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:45.048662 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebafb097-ceaa-43ca-9ce4-023cdca5ad56" path="/var/lib/kubelet/pods/ebafb097-ceaa-43ca-9ce4-023cdca5ad56/volumes" Oct 30 20:03:46.259159 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:46.259123 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722"] Oct 30 20:03:46.259616 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:46.259342 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" podUID="87cc3e48-0e55-4a0a-ac53-e586f8098a35" containerName="maas-api" containerID="cri-o://4711068e91739c83461729ebc55a4d25a4a793ced4ca68c259fe31fdd9a4f297" gracePeriod=30 Oct 30 20:03:46.538525 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:46.538488 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 20:03:46.639985 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:46.639950 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87cc3e48-0e55-4a0a-ac53-e586f8098a35-metrics-certs\" (UniqueName: \"kubernetes.io/secret/87cc3e48-0e55-4a0a-ac53-e586f8098a35-metrics-certs\") pod \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\" (UID: \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\") " Oct 30 20:03:46.640151 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:46.640068 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87cc3e48-0e55-4a0a-ac53-e586f8098a35-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87cc3e48-0e55-4a0a-ac53-e586f8098a35-maas-api-tls\") pod \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\" (UID: \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\") " Oct 30 20:03:46.640215 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:46.640146 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87cc3e48-0e55-4a0a-ac53-e586f8098a35-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87cc3e48-0e55-4a0a-ac53-e586f8098a35-openshift-service-ca\") pod \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\" (UID: \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\") " Oct 30 20:03:46.640215 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:46.640188 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87cc3e48-0e55-4a0a-ac53-e586f8098a35-kube-api-access-466dm\" (UniqueName: \"kubernetes.io/projected/87cc3e48-0e55-4a0a-ac53-e586f8098a35-kube-api-access-466dm\") pod \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\" (UID: \"87cc3e48-0e55-4a0a-ac53-e586f8098a35\") " Oct 30 20:03:46.640550 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:46.640518 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87cc3e48-0e55-4a0a-ac53-e586f8098a35-openshift-service-ca" pod "87cc3e48-0e55-4a0a-ac53-e586f8098a35" (UID: "87cc3e48-0e55-4a0a-ac53-e586f8098a35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:46.640662 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:46.640635 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87cc3e48-0e55-4a0a-ac53-e586f8098a35-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:46.642260 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:46.642233 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87cc3e48-0e55-4a0a-ac53-e586f8098a35-kube-api-access-466dm" pod "87cc3e48-0e55-4a0a-ac53-e586f8098a35" (UID: "87cc3e48-0e55-4a0a-ac53-e586f8098a35"). InnerVolumeSpecName "kube-api-access-466dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:46.642375 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:46.642342 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87cc3e48-0e55-4a0a-ac53-e586f8098a35-metrics-certs" pod "87cc3e48-0e55-4a0a-ac53-e586f8098a35" (UID: "87cc3e48-0e55-4a0a-ac53-e586f8098a35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:46.642638 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:46.642618 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87cc3e48-0e55-4a0a-ac53-e586f8098a35-maas-api-tls" pod "87cc3e48-0e55-4a0a-ac53-e586f8098a35" (UID: "87cc3e48-0e55-4a0a-ac53-e586f8098a35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:46.742188 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:46.742152 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87cc3e48-0e55-4a0a-ac53-e586f8098a35-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:46.742188 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:46.742189 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-466dm\" (UniqueName: \"kubernetes.io/projected/87cc3e48-0e55-4a0a-ac53-e586f8098a35-kube-api-access-466dm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:46.742396 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:46.742203 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87cc3e48-0e55-4a0a-ac53-e586f8098a35-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:47.279802 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:47.279766 5047 generic.go:356] "Generic (PLEG): container finished" podID="87cc3e48-0e55-4a0a-ac53-e586f8098a35" containerID="4711068e91739c83461729ebc55a4d25a4a793ced4ca68c259fe31fdd9a4f297" exitCode=0 Oct 30 20:03:47.280317 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:47.279817 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" Oct 30 20:03:47.280317 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:47.279854 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" event={"ID":"87cc3e48-0e55-4a0a-ac53-e586f8098a35","Type":"ContainerDied","Data":"4711068e91739c83461729ebc55a4d25a4a793ced4ca68c259fe31fdd9a4f297"} Oct 30 20:03:47.280317 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:47.279915 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722" event={"ID":"87cc3e48-0e55-4a0a-ac53-e586f8098a35","Type":"ContainerDied","Data":"78505714ccd9eec60d9333de953a3c6116aee3884aad1e4c8320d803af884154"} Oct 30 20:03:47.280317 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:47.279939 5047 scope.go:122] "RemoveContainer" containerID="4711068e91739c83461729ebc55a4d25a4a793ced4ca68c259fe31fdd9a4f297" Oct 30 20:03:47.289436 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:47.289408 5047 scope.go:122] "RemoveContainer" containerID="4711068e91739c83461729ebc55a4d25a4a793ced4ca68c259fe31fdd9a4f297" Oct 30 20:03:47.289725 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:03:47.289705 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4711068e91739c83461729ebc55a4d25a4a793ced4ca68c259fe31fdd9a4f297\": container with ID starting with 4711068e91739c83461729ebc55a4d25a4a793ced4ca68c259fe31fdd9a4f297 not found: ID does not exist" containerID="4711068e91739c83461729ebc55a4d25a4a793ced4ca68c259fe31fdd9a4f297" Oct 30 20:03:47.289784 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:47.289733 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4711068e91739c83461729ebc55a4d25a4a793ced4ca68c259fe31fdd9a4f297"} err="failed to get container status \"4711068e91739c83461729ebc55a4d25a4a793ced4ca68c259fe31fdd9a4f297\": rpc error: code = NotFound desc = could not find container \"4711068e91739c83461729ebc55a4d25a4a793ced4ca68c259fe31fdd9a4f297\": container with ID starting with 4711068e91739c83461729ebc55a4d25a4a793ced4ca68c259fe31fdd9a4f297 not found: ID does not exist" Oct 30 20:03:47.326222 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:47.326135 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722"] Oct 30 20:03:47.334376 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:47.334335 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-870193-5c489d55f8-tb722"] Oct 30 20:03:49.049136 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:49.049102 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87cc3e48-0e55-4a0a-ac53-e586f8098a35" path="/var/lib/kubelet/pods/87cc3e48-0e55-4a0a-ac53-e586f8098a35/volumes" Oct 30 20:03:55.103934 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:55.103863 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck"] Oct 30 20:03:55.105252 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:55.105214 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" podUID="9958d25e-765d-45db-9d7d-a2210b1903c9" containerName="maas-api" containerID="cri-o://9c924bab0017e3561f9ba3b477a73e9126a48049794ac752e631afad2fb3a8c7" gracePeriod=30 Oct 30 20:03:55.325558 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:55.325387 5047 generic.go:356] "Generic (PLEG): container finished" podID="9958d25e-765d-45db-9d7d-a2210b1903c9" containerID="9c924bab0017e3561f9ba3b477a73e9126a48049794ac752e631afad2fb3a8c7" exitCode=0 Oct 30 20:03:55.325829 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:55.325601 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" event={"ID":"9958d25e-765d-45db-9d7d-a2210b1903c9","Type":"ContainerDied","Data":"9c924bab0017e3561f9ba3b477a73e9126a48049794ac752e631afad2fb3a8c7"} Oct 30 20:03:55.380841 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:55.380811 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 20:03:55.433884 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:55.433837 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9958d25e-765d-45db-9d7d-a2210b1903c9-kube-api-access-vgfgx\" (UniqueName: \"kubernetes.io/projected/9958d25e-765d-45db-9d7d-a2210b1903c9-kube-api-access-vgfgx\") pod \"9958d25e-765d-45db-9d7d-a2210b1903c9\" (UID: \"9958d25e-765d-45db-9d7d-a2210b1903c9\") " Oct 30 20:03:55.434070 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:55.433942 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9958d25e-765d-45db-9d7d-a2210b1903c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9958d25e-765d-45db-9d7d-a2210b1903c9-maas-api-tls\") pod \"9958d25e-765d-45db-9d7d-a2210b1903c9\" (UID: \"9958d25e-765d-45db-9d7d-a2210b1903c9\") " Oct 30 20:03:55.434070 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:55.433982 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9958d25e-765d-45db-9d7d-a2210b1903c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9958d25e-765d-45db-9d7d-a2210b1903c9-openshift-service-ca\") pod \"9958d25e-765d-45db-9d7d-a2210b1903c9\" (UID: \"9958d25e-765d-45db-9d7d-a2210b1903c9\") " Oct 30 20:03:55.434270 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:55.434125 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9958d25e-765d-45db-9d7d-a2210b1903c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9958d25e-765d-45db-9d7d-a2210b1903c9-metrics-certs\") pod \"9958d25e-765d-45db-9d7d-a2210b1903c9\" (UID: \"9958d25e-765d-45db-9d7d-a2210b1903c9\") " Oct 30 20:03:55.434655 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:55.434464 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9958d25e-765d-45db-9d7d-a2210b1903c9-openshift-service-ca" pod "9958d25e-765d-45db-9d7d-a2210b1903c9" (UID: "9958d25e-765d-45db-9d7d-a2210b1903c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:55.436628 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:55.436601 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9958d25e-765d-45db-9d7d-a2210b1903c9-maas-api-tls" pod "9958d25e-765d-45db-9d7d-a2210b1903c9" (UID: "9958d25e-765d-45db-9d7d-a2210b1903c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:55.436749 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:55.436625 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9958d25e-765d-45db-9d7d-a2210b1903c9-kube-api-access-vgfgx" pod "9958d25e-765d-45db-9d7d-a2210b1903c9" (UID: "9958d25e-765d-45db-9d7d-a2210b1903c9"). InnerVolumeSpecName "kube-api-access-vgfgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:55.436876 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:55.436849 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9958d25e-765d-45db-9d7d-a2210b1903c9-metrics-certs" pod "9958d25e-765d-45db-9d7d-a2210b1903c9" (UID: "9958d25e-765d-45db-9d7d-a2210b1903c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:55.536592 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:55.536552 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9958d25e-765d-45db-9d7d-a2210b1903c9-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:55.536592 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:55.536585 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgfgx\" (UniqueName: \"kubernetes.io/projected/9958d25e-765d-45db-9d7d-a2210b1903c9-kube-api-access-vgfgx\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:55.536832 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:55.536600 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9958d25e-765d-45db-9d7d-a2210b1903c9-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:55.536832 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:55.536614 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9958d25e-765d-45db-9d7d-a2210b1903c9-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:03:56.332137 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:56.332105 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" Oct 30 20:03:56.332607 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:56.332100 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck" event={"ID":"9958d25e-765d-45db-9d7d-a2210b1903c9","Type":"ContainerDied","Data":"f4b9b4b161c5be98e7bd4be210230e79c1fb2b3ff8063b611d83e7f729ac2d88"} Oct 30 20:03:56.332607 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:56.332247 5047 scope.go:122] "RemoveContainer" containerID="9c924bab0017e3561f9ba3b477a73e9126a48049794ac752e631afad2fb3a8c7" Oct 30 20:03:56.377271 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:56.377229 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck"] Oct 30 20:03:56.387073 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:56.387036 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5a0a17-566c6f8644-vv2ck"] Oct 30 20:03:57.048770 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:03:57.048736 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9958d25e-765d-45db-9d7d-a2210b1903c9" path="/var/lib/kubelet/pods/9958d25e-765d-45db-9d7d-a2210b1903c9/volumes" Oct 30 20:04:03.592907 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:03.592870 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz"] Oct 30 20:04:03.593443 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:03.593167 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" podUID="f3712046-b84a-45fd-b20f-49c8d68adc11" containerName="istio-proxy" containerID="cri-o://22f1f7b47c15892b2f08ceeebb45fb5202c38a23386482dc3aeb100db1ad4b0c" gracePeriod=30 Oct 30 20:04:04.507671 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:04.507633 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t"] Oct 30 20:04:04.507919 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:04.507891 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" podUID="a05dd723-2d08-4261-aa55-e642430737e6" containerName="istio-proxy" containerID="cri-o://c80360e7660b98bb62fa9d4a2fd75c41716c4e035aedb99982847b68562af31b" gracePeriod=30 Oct 30 20:04:07.987864 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:07.987827 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 20:04:07.994108 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:07.994076 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 20:04:08.850948 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.850916 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 20:04:08.996884 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.996792 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-workload-socket\") pod \"f3712046-b84a-45fd-b20f-49c8d68adc11\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " Oct 30 20:04:08.996884 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.996838 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-token\" (UniqueName: \"kubernetes.io/projected/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-token\") pod \"f3712046-b84a-45fd-b20f-49c8d68adc11\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " Oct 30 20:04:08.996884 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.996859 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-data\") pod \"f3712046-b84a-45fd-b20f-49c8d68adc11\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " Oct 30 20:04:08.996884 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.996886 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-workload-certs\") pod \"f3712046-b84a-45fd-b20f-49c8d68adc11\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " Oct 30 20:04:08.997606 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.996927 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3712046-b84a-45fd-b20f-49c8d68adc11-kube-api-access-pznvl\" (UniqueName: \"kubernetes.io/projected/f3712046-b84a-45fd-b20f-49c8d68adc11-kube-api-access-pznvl\") pod \"f3712046-b84a-45fd-b20f-49c8d68adc11\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " Oct 30 20:04:08.997606 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.997061 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3712046-b84a-45fd-b20f-49c8d68adc11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3712046-b84a-45fd-b20f-49c8d68adc11-istiod-ca-cert\") pod \"f3712046-b84a-45fd-b20f-49c8d68adc11\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " Oct 30 20:04:08.997606 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.997101 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-credential-socket\") pod \"f3712046-b84a-45fd-b20f-49c8d68adc11\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " Oct 30 20:04:08.997606 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.997133 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-envoy\") pod \"f3712046-b84a-45fd-b20f-49c8d68adc11\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " Oct 30 20:04:08.997606 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.997211 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-podinfo\") pod \"f3712046-b84a-45fd-b20f-49c8d68adc11\" (UID: \"f3712046-b84a-45fd-b20f-49c8d68adc11\") " Oct 30 20:04:08.997606 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.997222 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-workload-socket" pod "f3712046-b84a-45fd-b20f-49c8d68adc11" (UID: "f3712046-b84a-45fd-b20f-49c8d68adc11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:08.997606 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.997232 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-data" pod "f3712046-b84a-45fd-b20f-49c8d68adc11" (UID: "f3712046-b84a-45fd-b20f-49c8d68adc11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:08.997606 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.997386 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-credential-socket" pod "f3712046-b84a-45fd-b20f-49c8d68adc11" (UID: "f3712046-b84a-45fd-b20f-49c8d68adc11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:08.997606 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.997498 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3712046-b84a-45fd-b20f-49c8d68adc11-istiod-ca-cert" pod "f3712046-b84a-45fd-b20f-49c8d68adc11" (UID: "f3712046-b84a-45fd-b20f-49c8d68adc11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:08.997606 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.997563 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:08.997606 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.997585 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:08.997606 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.997601 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:08.998401 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.997774 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-workload-certs" pod "f3712046-b84a-45fd-b20f-49c8d68adc11" (UID: "f3712046-b84a-45fd-b20f-49c8d68adc11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:08.999332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.999305 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-envoy" pod "f3712046-b84a-45fd-b20f-49c8d68adc11" (UID: "f3712046-b84a-45fd-b20f-49c8d68adc11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:08.999332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.999318 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-token" pod "f3712046-b84a-45fd-b20f-49c8d68adc11" (UID: "f3712046-b84a-45fd-b20f-49c8d68adc11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:08.999481 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.999371 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3712046-b84a-45fd-b20f-49c8d68adc11-kube-api-access-pznvl" pod "f3712046-b84a-45fd-b20f-49c8d68adc11" (UID: "f3712046-b84a-45fd-b20f-49c8d68adc11"). InnerVolumeSpecName "kube-api-access-pznvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:08.999548 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:08.999517 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-podinfo" pod "f3712046-b84a-45fd-b20f-49c8d68adc11" (UID: "f3712046-b84a-45fd-b20f-49c8d68adc11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:09.050285 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.050255 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4"] Oct 30 20:04:09.051472 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.051438 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6"] Oct 30 20:04:09.098179 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.098143 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pznvl\" (UniqueName: \"kubernetes.io/projected/f3712046-b84a-45fd-b20f-49c8d68adc11-kube-api-access-pznvl\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.098179 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.098170 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3712046-b84a-45fd-b20f-49c8d68adc11-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.098179 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.098180 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.098471 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.098189 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.098471 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.098198 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3712046-b84a-45fd-b20f-49c8d68adc11-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.098471 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.098206 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3712046-b84a-45fd-b20f-49c8d68adc11-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.413772 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.413740 5047 generic.go:356] "Generic (PLEG): container finished" podID="f3712046-b84a-45fd-b20f-49c8d68adc11" containerID="22f1f7b47c15892b2f08ceeebb45fb5202c38a23386482dc3aeb100db1ad4b0c" exitCode=0 Oct 30 20:04:09.413951 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.413804 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" Oct 30 20:04:09.413951 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.413826 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" event={"ID":"f3712046-b84a-45fd-b20f-49c8d68adc11","Type":"ContainerDied","Data":"22f1f7b47c15892b2f08ceeebb45fb5202c38a23386482dc3aeb100db1ad4b0c"} Oct 30 20:04:09.413951 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.413885 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz" event={"ID":"f3712046-b84a-45fd-b20f-49c8d68adc11","Type":"ContainerDied","Data":"0ec91d0363852b2f359dc12a0813ee3d24a6231a24a2fc567fe3afdcadd5bbd2"} Oct 30 20:04:09.413951 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.413914 5047 scope.go:122] "RemoveContainer" containerID="22f1f7b47c15892b2f08ceeebb45fb5202c38a23386482dc3aeb100db1ad4b0c" Oct 30 20:04:09.423682 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.423663 5047 scope.go:122] "RemoveContainer" containerID="64ce6678e99049f8c234669acbc47faf6ef5c2089a4f8c4b5806331dd25ede7b" Oct 30 20:04:09.432653 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.432634 5047 scope.go:122] "RemoveContainer" containerID="22f1f7b47c15892b2f08ceeebb45fb5202c38a23386482dc3aeb100db1ad4b0c" Oct 30 20:04:09.432933 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:04:09.432914 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22f1f7b47c15892b2f08ceeebb45fb5202c38a23386482dc3aeb100db1ad4b0c\": container with ID starting with 22f1f7b47c15892b2f08ceeebb45fb5202c38a23386482dc3aeb100db1ad4b0c not found: ID does not exist" containerID="22f1f7b47c15892b2f08ceeebb45fb5202c38a23386482dc3aeb100db1ad4b0c" Oct 30 20:04:09.433029 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.432944 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22f1f7b47c15892b2f08ceeebb45fb5202c38a23386482dc3aeb100db1ad4b0c"} err="failed to get container status \"22f1f7b47c15892b2f08ceeebb45fb5202c38a23386482dc3aeb100db1ad4b0c\": rpc error: code = NotFound desc = could not find container \"22f1f7b47c15892b2f08ceeebb45fb5202c38a23386482dc3aeb100db1ad4b0c\": container with ID starting with 22f1f7b47c15892b2f08ceeebb45fb5202c38a23386482dc3aeb100db1ad4b0c not found: ID does not exist" Oct 30 20:04:09.433029 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.432967 5047 scope.go:122] "RemoveContainer" containerID="64ce6678e99049f8c234669acbc47faf6ef5c2089a4f8c4b5806331dd25ede7b" Oct 30 20:04:09.433300 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:04:09.433277 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ce6678e99049f8c234669acbc47faf6ef5c2089a4f8c4b5806331dd25ede7b\": container with ID starting with 64ce6678e99049f8c234669acbc47faf6ef5c2089a4f8c4b5806331dd25ede7b not found: ID does not exist" containerID="64ce6678e99049f8c234669acbc47faf6ef5c2089a4f8c4b5806331dd25ede7b" Oct 30 20:04:09.433345 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.433306 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ce6678e99049f8c234669acbc47faf6ef5c2089a4f8c4b5806331dd25ede7b"} err="failed to get container status \"64ce6678e99049f8c234669acbc47faf6ef5c2089a4f8c4b5806331dd25ede7b\": rpc error: code = NotFound desc = could not find container \"64ce6678e99049f8c234669acbc47faf6ef5c2089a4f8c4b5806331dd25ede7b\": container with ID starting with 64ce6678e99049f8c234669acbc47faf6ef5c2089a4f8c4b5806331dd25ede7b not found: ID does not exist" Oct 30 20:04:09.460146 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.460119 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz"] Oct 30 20:04:09.465555 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.465532 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5a0a17-openshift-default-75d6966497-2bgjz"] Oct 30 20:04:09.600932 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:04:09.600897 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:04:09.766316 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.766279 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 20:04:09.906334 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.906294 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-istio-envoy\") pod \"a05dd723-2d08-4261-aa55-e642430737e6\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " Oct 30 20:04:09.906523 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.906389 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05dd723-2d08-4261-aa55-e642430737e6-kube-api-access-dn7vk\" (UniqueName: \"kubernetes.io/projected/a05dd723-2d08-4261-aa55-e642430737e6-kube-api-access-dn7vk\") pod \"a05dd723-2d08-4261-aa55-e642430737e6\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " Oct 30 20:04:09.906523 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.906427 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-workload-socket\") pod \"a05dd723-2d08-4261-aa55-e642430737e6\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " Oct 30 20:04:09.906523 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.906455 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-credential-socket\") pod \"a05dd723-2d08-4261-aa55-e642430737e6\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " Oct 30 20:04:09.906523 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.906490 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-istio-data\") pod \"a05dd723-2d08-4261-aa55-e642430737e6\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " Oct 30 20:04:09.906523 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.906514 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05dd723-2d08-4261-aa55-e642430737e6-istio-token\" (UniqueName: \"kubernetes.io/projected/a05dd723-2d08-4261-aa55-e642430737e6-istio-token\") pod \"a05dd723-2d08-4261-aa55-e642430737e6\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " Oct 30 20:04:09.906819 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.906559 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-workload-certs\") pod \"a05dd723-2d08-4261-aa55-e642430737e6\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " Oct 30 20:04:09.906819 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.906595 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a05dd723-2d08-4261-aa55-e642430737e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05dd723-2d08-4261-aa55-e642430737e6-istio-podinfo\") pod \"a05dd723-2d08-4261-aa55-e642430737e6\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " Oct 30 20:04:09.906819 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.906655 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a05dd723-2d08-4261-aa55-e642430737e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05dd723-2d08-4261-aa55-e642430737e6-istiod-ca-cert\") pod \"a05dd723-2d08-4261-aa55-e642430737e6\" (UID: \"a05dd723-2d08-4261-aa55-e642430737e6\") " Oct 30 20:04:09.906819 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.906799 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-istio-data" pod "a05dd723-2d08-4261-aa55-e642430737e6" (UID: "a05dd723-2d08-4261-aa55-e642430737e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:09.907060 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.906813 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-workload-socket" pod "a05dd723-2d08-4261-aa55-e642430737e6" (UID: "a05dd723-2d08-4261-aa55-e642430737e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:09.907060 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.906832 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-credential-socket" pod "a05dd723-2d08-4261-aa55-e642430737e6" (UID: "a05dd723-2d08-4261-aa55-e642430737e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:09.907060 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.906925 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-workload-certs" pod "a05dd723-2d08-4261-aa55-e642430737e6" (UID: "a05dd723-2d08-4261-aa55-e642430737e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:09.907217 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.907121 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05dd723-2d08-4261-aa55-e642430737e6-istiod-ca-cert" pod "a05dd723-2d08-4261-aa55-e642430737e6" (UID: "a05dd723-2d08-4261-aa55-e642430737e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:09.907217 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.907142 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.907217 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.907161 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.907217 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.907174 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.907217 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.907184 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:09.908916 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.908887 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a05dd723-2d08-4261-aa55-e642430737e6-istio-podinfo" pod "a05dd723-2d08-4261-aa55-e642430737e6" (UID: "a05dd723-2d08-4261-aa55-e642430737e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:09.908916 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.908892 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05dd723-2d08-4261-aa55-e642430737e6-kube-api-access-dn7vk" pod "a05dd723-2d08-4261-aa55-e642430737e6" (UID: "a05dd723-2d08-4261-aa55-e642430737e6"). InnerVolumeSpecName "kube-api-access-dn7vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:09.909118 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.909025 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-istio-envoy" pod "a05dd723-2d08-4261-aa55-e642430737e6" (UID: "a05dd723-2d08-4261-aa55-e642430737e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:09.909118 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:09.909042 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05dd723-2d08-4261-aa55-e642430737e6-istio-token" pod "a05dd723-2d08-4261-aa55-e642430737e6" (UID: "a05dd723-2d08-4261-aa55-e642430737e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:10.008604 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:10.008513 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05dd723-2d08-4261-aa55-e642430737e6-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:10.008604 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:10.008555 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn7vk\" (UniqueName: \"kubernetes.io/projected/a05dd723-2d08-4261-aa55-e642430737e6-kube-api-access-dn7vk\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:10.008604 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:10.008566 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05dd723-2d08-4261-aa55-e642430737e6-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:10.008604 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:10.008575 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05dd723-2d08-4261-aa55-e642430737e6-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:10.008604 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:10.008583 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05dd723-2d08-4261-aa55-e642430737e6-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:10.421463 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:10.421428 5047 generic.go:356] "Generic (PLEG): container finished" podID="a05dd723-2d08-4261-aa55-e642430737e6" containerID="c80360e7660b98bb62fa9d4a2fd75c41716c4e035aedb99982847b68562af31b" exitCode=0 Oct 30 20:04:10.421689 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:10.421481 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" Oct 30 20:04:10.421689 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:10.421517 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" event={"ID":"a05dd723-2d08-4261-aa55-e642430737e6","Type":"ContainerDied","Data":"c80360e7660b98bb62fa9d4a2fd75c41716c4e035aedb99982847b68562af31b"} Oct 30 20:04:10.421689 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:10.421583 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t" event={"ID":"a05dd723-2d08-4261-aa55-e642430737e6","Type":"ContainerDied","Data":"832eb5e97f58ed84312a98cff121694fab81f71310da7ecf0f7ec40bcfe28b5e"} Oct 30 20:04:10.421689 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:10.421613 5047 scope.go:122] "RemoveContainer" containerID="c80360e7660b98bb62fa9d4a2fd75c41716c4e035aedb99982847b68562af31b" Oct 30 20:04:10.431930 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:10.431912 5047 scope.go:122] "RemoveContainer" containerID="854d2769d396b31c54c5370335d46309f17672d4c1b0dbd59b793ab0e5c754a9" Oct 30 20:04:10.440908 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:10.440887 5047 scope.go:122] "RemoveContainer" containerID="c80360e7660b98bb62fa9d4a2fd75c41716c4e035aedb99982847b68562af31b" Oct 30 20:04:10.441190 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:04:10.441166 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80360e7660b98bb62fa9d4a2fd75c41716c4e035aedb99982847b68562af31b\": container with ID starting with c80360e7660b98bb62fa9d4a2fd75c41716c4e035aedb99982847b68562af31b not found: ID does not exist" containerID="c80360e7660b98bb62fa9d4a2fd75c41716c4e035aedb99982847b68562af31b" Oct 30 20:04:10.441256 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:10.441202 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80360e7660b98bb62fa9d4a2fd75c41716c4e035aedb99982847b68562af31b"} err="failed to get container status \"c80360e7660b98bb62fa9d4a2fd75c41716c4e035aedb99982847b68562af31b\": rpc error: code = NotFound desc = could not find container \"c80360e7660b98bb62fa9d4a2fd75c41716c4e035aedb99982847b68562af31b\": container with ID starting with c80360e7660b98bb62fa9d4a2fd75c41716c4e035aedb99982847b68562af31b not found: ID does not exist" Oct 30 20:04:10.441256 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:10.441222 5047 scope.go:122] "RemoveContainer" containerID="854d2769d396b31c54c5370335d46309f17672d4c1b0dbd59b793ab0e5c754a9" Oct 30 20:04:10.441465 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:04:10.441447 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854d2769d396b31c54c5370335d46309f17672d4c1b0dbd59b793ab0e5c754a9\": container with ID starting with 854d2769d396b31c54c5370335d46309f17672d4c1b0dbd59b793ab0e5c754a9 not found: ID does not exist" containerID="854d2769d396b31c54c5370335d46309f17672d4c1b0dbd59b793ab0e5c754a9" Oct 30 20:04:10.441503 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:10.441470 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854d2769d396b31c54c5370335d46309f17672d4c1b0dbd59b793ab0e5c754a9"} err="failed to get container status \"854d2769d396b31c54c5370335d46309f17672d4c1b0dbd59b793ab0e5c754a9\": rpc error: code = NotFound desc = could not find container \"854d2769d396b31c54c5370335d46309f17672d4c1b0dbd59b793ab0e5c754a9\": container with ID starting with 854d2769d396b31c54c5370335d46309f17672d4c1b0dbd59b793ab0e5c754a9 not found: ID does not exist" Oct 30 20:04:10.466822 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:10.466790 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t"] Oct 30 20:04:10.476734 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:10.476703 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-870193-openshift-default-6bdcd57969-kl42t"] Oct 30 20:04:11.047836 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:11.047802 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a05dd723-2d08-4261-aa55-e642430737e6" path="/var/lib/kubelet/pods/a05dd723-2d08-4261-aa55-e642430737e6/volumes" Oct 30 20:04:11.048329 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:11.048312 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3712046-b84a-45fd-b20f-49c8d68adc11" path="/var/lib/kubelet/pods/f3712046-b84a-45fd-b20f-49c8d68adc11/volumes" Oct 30 20:04:17.321989 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.321950 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn"] Oct 30 20:04:17.322601 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322579 5047 cpu_manager.go:414] "containerMap: removing container" podUID="2123dbbb-db97-46f3-b265-a81414f0652c" containerName="istio-proxy" Oct 30 20:04:17.322700 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322603 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2123dbbb-db97-46f3-b265-a81414f0652c" containerName="istio-proxy" Oct 30 20:04:17.322700 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322617 5047 cpu_manager.go:414] "containerMap: removing container" podUID="f3712046-b84a-45fd-b20f-49c8d68adc11" containerName="istio-proxy" Oct 30 20:04:17.322700 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322626 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3712046-b84a-45fd-b20f-49c8d68adc11" containerName="istio-proxy" Oct 30 20:04:17.322700 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322643 5047 cpu_manager.go:414] "containerMap: removing container" podUID="64a39903-20d2-4c30-88a0-57cf7c222212" containerName="payload-processing" Oct 30 20:04:17.322700 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322651 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a39903-20d2-4c30-88a0-57cf7c222212" containerName="payload-processing" Oct 30 20:04:17.322700 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322660 5047 cpu_manager.go:414] "containerMap: removing container" podUID="52b11843-e441-4d53-a386-bc1b63404d00" containerName="maas-api" Oct 30 20:04:17.322700 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322667 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b11843-e441-4d53-a386-bc1b63404d00" containerName="maas-api" Oct 30 20:04:17.322700 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322686 5047 cpu_manager.go:414] "containerMap: removing container" podUID="f3712046-b84a-45fd-b20f-49c8d68adc11" containerName="istio-proxy" Oct 30 20:04:17.322700 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322694 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3712046-b84a-45fd-b20f-49c8d68adc11" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322707 5047 cpu_manager.go:414] "containerMap: removing container" podUID="ebafb097-ceaa-43ca-9ce4-023cdca5ad56" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322715 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebafb097-ceaa-43ca-9ce4-023cdca5ad56" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322726 5047 cpu_manager.go:414] "containerMap: removing container" podUID="d764690a-8072-41fb-b021-1c6ad226146f" containerName="payload-pre-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322735 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d764690a-8072-41fb-b021-1c6ad226146f" containerName="payload-pre-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322748 5047 cpu_manager.go:414] "containerMap: removing container" podUID="ce2235f8-00fb-44bc-90f8-6629cfc22510" containerName="maas-api" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322755 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2235f8-00fb-44bc-90f8-6629cfc22510" containerName="maas-api" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322772 5047 cpu_manager.go:414] "containerMap: removing container" podUID="87b98550-4b44-4b24-8c55-03a8e05f08df" containerName="payload-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322780 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b98550-4b44-4b24-8c55-03a8e05f08df" containerName="payload-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322798 5047 cpu_manager.go:414] "containerMap: removing container" podUID="d1e68671-670d-4915-ba4d-47c184994e6a" containerName="payload-pre-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322805 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e68671-670d-4915-ba4d-47c184994e6a" containerName="payload-pre-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322817 5047 cpu_manager.go:414] "containerMap: removing container" podUID="845e76e9-51c7-440d-a89e-60adc522d008" containerName="payload-pre-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322824 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845e76e9-51c7-440d-a89e-60adc522d008" containerName="payload-pre-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322835 5047 cpu_manager.go:414] "containerMap: removing container" podUID="ebafb097-ceaa-43ca-9ce4-023cdca5ad56" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322842 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebafb097-ceaa-43ca-9ce4-023cdca5ad56" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322854 5047 cpu_manager.go:414] "containerMap: removing container" podUID="4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0" containerName="payload-pre-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322861 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0" containerName="payload-pre-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322871 5047 cpu_manager.go:414] "containerMap: removing container" podUID="aa3c8991-f71f-4500-b4b7-e25088017a29" containerName="payload-pre-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322878 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa3c8991-f71f-4500-b4b7-e25088017a29" containerName="payload-pre-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322890 5047 cpu_manager.go:414] "containerMap: removing container" podUID="bfef2562-28c6-4640-a1c0-8a467bba8a79" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322897 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfef2562-28c6-4640-a1c0-8a467bba8a79" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322911 5047 cpu_manager.go:414] "containerMap: removing container" podUID="5a2dff4c-72fb-4569-a49a-f8995370ad75" containerName="maas-api" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322918 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a2dff4c-72fb-4569-a49a-f8995370ad75" containerName="maas-api" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322978 5047 cpu_manager.go:414] "containerMap: removing container" podUID="ea906da9-9a7c-4755-b5f1-2f6388875be2" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.322988 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea906da9-9a7c-4755-b5f1-2f6388875be2" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323000 5047 cpu_manager.go:414] "containerMap: removing container" podUID="e1f14230-b23c-42c4-9709-f67299e49c68" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323021 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f14230-b23c-42c4-9709-f67299e49c68" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323036 5047 cpu_manager.go:414] "containerMap: removing container" podUID="b913225f-a9cf-4af0-af6f-076a9c0ec683" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323043 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b913225f-a9cf-4af0-af6f-076a9c0ec683" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323055 5047 cpu_manager.go:414] "containerMap: removing container" podUID="a12a3bdc-6b65-4496-8458-ff694a332732" containerName="maas-api" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323063 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a12a3bdc-6b65-4496-8458-ff694a332732" containerName="maas-api" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323076 5047 cpu_manager.go:414] "containerMap: removing container" podUID="ea906da9-9a7c-4755-b5f1-2f6388875be2" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323083 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea906da9-9a7c-4755-b5f1-2f6388875be2" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323099 5047 cpu_manager.go:414] "containerMap: removing container" podUID="71ff13b0-8eb3-474b-8579-29b192ccd9ee" containerName="payload-pre-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323106 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ff13b0-8eb3-474b-8579-29b192ccd9ee" containerName="payload-pre-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323122 5047 cpu_manager.go:414] "containerMap: removing container" podUID="2ac686d8-338e-43b4-81b0-15718ed61608" containerName="payload-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323130 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ac686d8-338e-43b4-81b0-15718ed61608" containerName="payload-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323147 5047 cpu_manager.go:414] "containerMap: removing container" podUID="2123dbbb-db97-46f3-b265-a81414f0652c" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323155 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2123dbbb-db97-46f3-b265-a81414f0652c" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323165 5047 cpu_manager.go:414] "containerMap: removing container" podUID="87cc3e48-0e55-4a0a-ac53-e586f8098a35" containerName="maas-api" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323172 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87cc3e48-0e55-4a0a-ac53-e586f8098a35" containerName="maas-api" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323184 5047 cpu_manager.go:414] "containerMap: removing container" podUID="a05dd723-2d08-4261-aa55-e642430737e6" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323191 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05dd723-2d08-4261-aa55-e642430737e6" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323201 5047 cpu_manager.go:414] "containerMap: removing container" podUID="a05dd723-2d08-4261-aa55-e642430737e6" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323208 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05dd723-2d08-4261-aa55-e642430737e6" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323219 5047 cpu_manager.go:414] "containerMap: removing container" podUID="3e35030c-45c1-4c40-98e8-8a4a1fe5b72f" containerName="payload-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323225 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e35030c-45c1-4c40-98e8-8a4a1fe5b72f" containerName="payload-processing" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323243 5047 cpu_manager.go:414] "containerMap: removing container" podUID="89b442c9-df9e-42fb-a944-5b1ac933efa4" containerName="maas-api" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323251 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b442c9-df9e-42fb-a944-5b1ac933efa4" containerName="maas-api" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323263 5047 cpu_manager.go:414] "containerMap: removing container" podUID="c0cd76c6-9b67-4719-9d13-82315cd9f77a" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323271 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0cd76c6-9b67-4719-9d13-82315cd9f77a" containerName="istio-proxy" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323283 5047 cpu_manager.go:414] "containerMap: removing container" podUID="3d40dfe2-d6ef-449d-a678-8e6938ea132e" containerName="maas-api" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323290 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d40dfe2-d6ef-449d-a678-8e6938ea132e" containerName="maas-api" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323299 5047 cpu_manager.go:414] "containerMap: removing container" podUID="9958d25e-765d-45db-9d7d-a2210b1903c9" containerName="maas-api" Oct 30 20:04:17.323284 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323306 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9958d25e-765d-45db-9d7d-a2210b1903c9" containerName="maas-api" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323319 5047 cpu_manager.go:414] "containerMap: removing container" podUID="d4b63ca7-418b-44c0-a603-575c14f97efb" containerName="maas-api" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323326 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b63ca7-418b-44c0-a603-575c14f97efb" containerName="maas-api" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323339 5047 cpu_manager.go:414] "containerMap: removing container" podUID="0c851817-5127-4d01-91db-5f6bf9e8a062" containerName="payload-processing" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323348 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c851817-5127-4d01-91db-5f6bf9e8a062" containerName="payload-processing" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323476 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="a12a3bdc-6b65-4496-8458-ff694a332732" containerName="maas-api" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323493 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea906da9-9a7c-4755-b5f1-2f6388875be2" containerName="istio-proxy" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323505 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ff13b0-8eb3-474b-8579-29b192ccd9ee" containerName="payload-pre-processing" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323516 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ac686d8-338e-43b4-81b0-15718ed61608" containerName="payload-processing" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323527 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="2123dbbb-db97-46f3-b265-a81414f0652c" containerName="istio-proxy" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323536 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="87cc3e48-0e55-4a0a-ac53-e586f8098a35" containerName="maas-api" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323545 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="a05dd723-2d08-4261-aa55-e642430737e6" containerName="istio-proxy" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323554 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="a05dd723-2d08-4261-aa55-e642430737e6" containerName="istio-proxy" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323563 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e35030c-45c1-4c40-98e8-8a4a1fe5b72f" containerName="payload-processing" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323572 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b442c9-df9e-42fb-a944-5b1ac933efa4" containerName="maas-api" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323581 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0cd76c6-9b67-4719-9d13-82315cd9f77a" containerName="istio-proxy" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323588 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d40dfe2-d6ef-449d-a678-8e6938ea132e" containerName="maas-api" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323598 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="9958d25e-765d-45db-9d7d-a2210b1903c9" containerName="maas-api" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323606 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b63ca7-418b-44c0-a603-575c14f97efb" containerName="maas-api" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323645 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c851817-5127-4d01-91db-5f6bf9e8a062" containerName="payload-processing" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323656 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="2123dbbb-db97-46f3-b265-a81414f0652c" containerName="istio-proxy" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323665 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3712046-b84a-45fd-b20f-49c8d68adc11" containerName="istio-proxy" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323678 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a39903-20d2-4c30-88a0-57cf7c222212" containerName="payload-processing" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323686 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b11843-e441-4d53-a386-bc1b63404d00" containerName="maas-api" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323696 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3712046-b84a-45fd-b20f-49c8d68adc11" containerName="istio-proxy" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323704 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebafb097-ceaa-43ca-9ce4-023cdca5ad56" containerName="istio-proxy" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323714 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="d764690a-8072-41fb-b021-1c6ad226146f" containerName="payload-pre-processing" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323724 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2235f8-00fb-44bc-90f8-6629cfc22510" containerName="maas-api" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323731 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b98550-4b44-4b24-8c55-03a8e05f08df" containerName="payload-processing" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323742 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e68671-670d-4915-ba4d-47c184994e6a" containerName="payload-pre-processing" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323753 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="845e76e9-51c7-440d-a89e-60adc522d008" containerName="payload-pre-processing" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323764 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebafb097-ceaa-43ca-9ce4-023cdca5ad56" containerName="istio-proxy" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323774 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fbc3d85-8a2c-4321-9f83-5e250e7c6eb0" containerName="payload-pre-processing" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323782 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa3c8991-f71f-4500-b4b7-e25088017a29" containerName="payload-pre-processing" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323792 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfef2562-28c6-4640-a1c0-8a467bba8a79" containerName="istio-proxy" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323802 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a2dff4c-72fb-4569-a49a-f8995370ad75" containerName="maas-api" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323812 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f14230-b23c-42c4-9709-f67299e49c68" containerName="istio-proxy" Oct 30 20:04:17.326098 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.323822 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="b913225f-a9cf-4af0-af6f-076a9c0ec683" containerName="istio-proxy" Oct 30 20:04:17.328893 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.328861 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" Oct 30 20:04:17.347030 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.346973 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn"] Oct 30 20:04:17.354573 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.354539 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:04:17.354737 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.354695 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp88b\"" Oct 30 20:04:17.379743 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.379715 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032d435a-1b1a-4bd4-acd6-968efd75afa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-168c11f5-2b6mn\" (UID: \"032d435a-1b1a-4bd4-acd6-968efd75afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" Oct 30 20:04:17.379903 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.379758 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032d435a-1b1a-4bd4-acd6-968efd75afa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-168c11f5-2b6mn\" (UID: \"032d435a-1b1a-4bd4-acd6-968efd75afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" Oct 30 20:04:17.379903 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.379786 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmprh\" (UniqueName: \"kubernetes.io/projected/032d435a-1b1a-4bd4-acd6-968efd75afa4-kube-api-access-cmprh\") pod \"maas-api-revoke-sub-168c11f5-2b6mn\" (UID: \"032d435a-1b1a-4bd4-acd6-968efd75afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" Oct 30 20:04:17.480999 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.480958 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032d435a-1b1a-4bd4-acd6-968efd75afa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-168c11f5-2b6mn\" (UID: \"032d435a-1b1a-4bd4-acd6-968efd75afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" Oct 30 20:04:17.481217 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.481035 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032d435a-1b1a-4bd4-acd6-968efd75afa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-168c11f5-2b6mn\" (UID: \"032d435a-1b1a-4bd4-acd6-968efd75afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" Oct 30 20:04:17.481217 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.481067 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmprh\" (UniqueName: \"kubernetes.io/projected/032d435a-1b1a-4bd4-acd6-968efd75afa4-kube-api-access-cmprh\") pod \"maas-api-revoke-sub-168c11f5-2b6mn\" (UID: \"032d435a-1b1a-4bd4-acd6-968efd75afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" Oct 30 20:04:17.481836 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.481809 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032d435a-1b1a-4bd4-acd6-968efd75afa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-168c11f5-2b6mn\" (UID: \"032d435a-1b1a-4bd4-acd6-968efd75afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" Oct 30 20:04:17.483585 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.483565 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032d435a-1b1a-4bd4-acd6-968efd75afa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-168c11f5-2b6mn\" (UID: \"032d435a-1b1a-4bd4-acd6-968efd75afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" Oct 30 20:04:17.498920 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.498889 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmprh\" (UniqueName: \"kubernetes.io/projected/032d435a-1b1a-4bd4-acd6-968efd75afa4-kube-api-access-cmprh\") pod \"maas-api-revoke-sub-168c11f5-2b6mn\" (UID: \"032d435a-1b1a-4bd4-acd6-968efd75afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" Oct 30 20:04:17.644949 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.644910 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" Oct 30 20:04:17.947360 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:17.947331 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn"] Oct 30 20:04:17.949320 ip-10-0-140-235 kubenswrapper[5047]: W1030 20:04:17.949289 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032d435a_1b1a_4bd4_acd6_968efd75afa4.slice/crio-2eb5cf54d17f0017379d4832b6f784e1a0321f18cd4e0ea06f509d4076d83e0f WatchSource:0}: Error finding container 2eb5cf54d17f0017379d4832b6f784e1a0321f18cd4e0ea06f509d4076d83e0f: Status 404 returned error can't find the container with id 2eb5cf54d17f0017379d4832b6f784e1a0321f18cd4e0ea06f509d4076d83e0f Oct 30 20:04:18.467866 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:18.467828 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" event={"ID":"032d435a-1b1a-4bd4-acd6-968efd75afa4","Type":"ContainerStarted","Data":"180456f5d4efd080f4e155baec917c381194f530a3099162106b9a6f688b9aaf"} Oct 30 20:04:18.468291 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:18.467885 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" event={"ID":"032d435a-1b1a-4bd4-acd6-968efd75afa4","Type":"ContainerStarted","Data":"2eb5cf54d17f0017379d4832b6f784e1a0321f18cd4e0ea06f509d4076d83e0f"} Oct 30 20:04:18.509449 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:18.509371 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" podStartSLOduration=1.5093601909999999 podStartE2EDuration="1.509360191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:18.50675844 +0000 UTC m=+3611.581213799" watchObservedRunningTime="2026-10-30 20:04:18.509360191 +0000 UTC m=+3611.583815585" Oct 30 20:04:19.475064 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:19.475022 5047 generic.go:356] "Generic (PLEG): container finished" podID="032d435a-1b1a-4bd4-acd6-968efd75afa4" containerID="180456f5d4efd080f4e155baec917c381194f530a3099162106b9a6f688b9aaf" exitCode=0 Oct 30 20:04:19.475541 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:19.475111 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" event={"ID":"032d435a-1b1a-4bd4-acd6-968efd75afa4","Type":"ContainerDied","Data":"180456f5d4efd080f4e155baec917c381194f530a3099162106b9a6f688b9aaf"} Oct 30 20:04:20.622731 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:20.622702 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" Oct 30 20:04:20.712780 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:20.712731 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/032d435a-1b1a-4bd4-acd6-968efd75afa4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032d435a-1b1a-4bd4-acd6-968efd75afa4-maas-api-xks-ca\") pod \"032d435a-1b1a-4bd4-acd6-968efd75afa4\" (UID: \"032d435a-1b1a-4bd4-acd6-968efd75afa4\") " Oct 30 20:04:20.712780 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:20.712781 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/032d435a-1b1a-4bd4-acd6-968efd75afa4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032d435a-1b1a-4bd4-acd6-968efd75afa4-maas-api-openshift-ca\") pod \"032d435a-1b1a-4bd4-acd6-968efd75afa4\" (UID: \"032d435a-1b1a-4bd4-acd6-968efd75afa4\") " Oct 30 20:04:20.713039 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:20.712831 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/032d435a-1b1a-4bd4-acd6-968efd75afa4-kube-api-access-cmprh\" (UniqueName: \"kubernetes.io/projected/032d435a-1b1a-4bd4-acd6-968efd75afa4-kube-api-access-cmprh\") pod \"032d435a-1b1a-4bd4-acd6-968efd75afa4\" (UID: \"032d435a-1b1a-4bd4-acd6-968efd75afa4\") " Oct 30 20:04:20.713271 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:20.713241 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032d435a-1b1a-4bd4-acd6-968efd75afa4-maas-api-openshift-ca" pod "032d435a-1b1a-4bd4-acd6-968efd75afa4" (UID: "032d435a-1b1a-4bd4-acd6-968efd75afa4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:20.715089 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:20.715058 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/032d435a-1b1a-4bd4-acd6-968efd75afa4-maas-api-xks-ca" pod "032d435a-1b1a-4bd4-acd6-968efd75afa4" (UID: "032d435a-1b1a-4bd4-acd6-968efd75afa4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:20.715210 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:20.715130 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032d435a-1b1a-4bd4-acd6-968efd75afa4-kube-api-access-cmprh" pod "032d435a-1b1a-4bd4-acd6-968efd75afa4" (UID: "032d435a-1b1a-4bd4-acd6-968efd75afa4"). InnerVolumeSpecName "kube-api-access-cmprh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:20.814572 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:20.814481 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032d435a-1b1a-4bd4-acd6-968efd75afa4-maas-api-xks-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.814572 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:20.814516 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032d435a-1b1a-4bd4-acd6-968efd75afa4-maas-api-openshift-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.814572 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:20.814526 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmprh\" (UniqueName: \"kubernetes.io/projected/032d435a-1b1a-4bd4-acd6-968efd75afa4-kube-api-access-cmprh\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:21.485583 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:21.485555 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" Oct 30 20:04:21.485750 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:21.485558 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn" event={"ID":"032d435a-1b1a-4bd4-acd6-968efd75afa4","Type":"ContainerDied","Data":"2eb5cf54d17f0017379d4832b6f784e1a0321f18cd4e0ea06f509d4076d83e0f"} Oct 30 20:04:21.485750 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:21.485680 5047 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eb5cf54d17f0017379d4832b6f784e1a0321f18cd4e0ea06f509d4076d83e0f" Oct 30 20:04:24.072412 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.072372 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" containerID="cri-o://714a7ead6ec214e8ddc5ec5e619959eb3c8d4fb4caf8562196648361dc820b89" gracePeriod=45 Oct 30 20:04:24.072876 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.072671 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" containerID="cri-o://3c5a3b406ee4b4e127256a19241ec5d9a97266a91c40e1e2b9aafc37a3689c6b" gracePeriod=45 Oct 30 20:04:24.356472 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.356447 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 20:04:24.359765 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.359746 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 20:04:24.451917 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.451876 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-home\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-home\") pod \"55ce8a09-3219-4883-adf2-452d0dd05b18\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " Oct 30 20:04:24.451917 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.451919 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55ce8a09-3219-4883-adf2-452d0dd05b18-tls-certs\" (UniqueName: \"kubernetes.io/secret/55ce8a09-3219-4883-adf2-452d0dd05b18-tls-certs\") pod \"55ce8a09-3219-4883-adf2-452d0dd05b18\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " Oct 30 20:04:24.452195 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.451976 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-tmp-dir\") pod \"d6339b71-2746-47d8-810b-217e48b23e0e\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " Oct 30 20:04:24.452195 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452024 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-model-cache\") pod \"d6339b71-2746-47d8-810b-217e48b23e0e\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " Oct 30 20:04:24.452195 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452138 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ce8a09-3219-4883-adf2-452d0dd05b18-kube-api-access-gcz5q\" (UniqueName: \"kubernetes.io/projected/55ce8a09-3219-4883-adf2-452d0dd05b18-kube-api-access-gcz5q\") pod \"55ce8a09-3219-4883-adf2-452d0dd05b18\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " Oct 30 20:04:24.452195 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452180 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-dshm\") pod \"55ce8a09-3219-4883-adf2-452d0dd05b18\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " Oct 30 20:04:24.452437 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452214 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-home" pod "55ce8a09-3219-4883-adf2-452d0dd05b18" (UID: "55ce8a09-3219-4883-adf2-452d0dd05b18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:24.452437 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452223 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-home\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-home\") pod \"d6339b71-2746-47d8-810b-217e48b23e0e\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " Oct 30 20:04:24.452437 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452248 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-model-cache" pod "d6339b71-2746-47d8-810b-217e48b23e0e" (UID: "d6339b71-2746-47d8-810b-217e48b23e0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:24.452437 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452262 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6339b71-2746-47d8-810b-217e48b23e0e-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6339b71-2746-47d8-810b-217e48b23e0e-tls-certs\") pod \"d6339b71-2746-47d8-810b-217e48b23e0e\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " Oct 30 20:04:24.452437 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452298 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-dshm\") pod \"d6339b71-2746-47d8-810b-217e48b23e0e\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " Oct 30 20:04:24.452437 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452325 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-tmp-dir" pod "d6339b71-2746-47d8-810b-217e48b23e0e" (UID: "d6339b71-2746-47d8-810b-217e48b23e0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:24.452437 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452350 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-model-cache\") pod \"55ce8a09-3219-4883-adf2-452d0dd05b18\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " Oct 30 20:04:24.452437 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452389 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6339b71-2746-47d8-810b-217e48b23e0e-kube-api-access-mchkj\" (UniqueName: \"kubernetes.io/projected/d6339b71-2746-47d8-810b-217e48b23e0e-kube-api-access-mchkj\") pod \"d6339b71-2746-47d8-810b-217e48b23e0e\" (UID: \"d6339b71-2746-47d8-810b-217e48b23e0e\") " Oct 30 20:04:24.452437 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452441 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-tmp-dir\") pod \"55ce8a09-3219-4883-adf2-452d0dd05b18\" (UID: \"55ce8a09-3219-4883-adf2-452d0dd05b18\") " Oct 30 20:04:24.452945 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452760 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-home" pod "d6339b71-2746-47d8-810b-217e48b23e0e" (UID: "d6339b71-2746-47d8-810b-217e48b23e0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:24.452945 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452880 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.452945 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452904 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.452945 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452918 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.452945 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452931 5047 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-home\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.453253 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.452972 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-model-cache" pod "55ce8a09-3219-4883-adf2-452d0dd05b18" (UID: "55ce8a09-3219-4883-adf2-452d0dd05b18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:24.453517 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.453490 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-tmp-dir" pod "55ce8a09-3219-4883-adf2-452d0dd05b18" (UID: "55ce8a09-3219-4883-adf2-452d0dd05b18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:24.455658 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.455628 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-dshm" pod "d6339b71-2746-47d8-810b-217e48b23e0e" (UID: "d6339b71-2746-47d8-810b-217e48b23e0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:24.455801 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.455699 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6339b71-2746-47d8-810b-217e48b23e0e-tls-certs" pod "d6339b71-2746-47d8-810b-217e48b23e0e" (UID: "d6339b71-2746-47d8-810b-217e48b23e0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:24.456202 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.456178 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6339b71-2746-47d8-810b-217e48b23e0e-kube-api-access-mchkj" pod "d6339b71-2746-47d8-810b-217e48b23e0e" (UID: "d6339b71-2746-47d8-810b-217e48b23e0e"). InnerVolumeSpecName "kube-api-access-mchkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:24.456295 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.456196 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ce8a09-3219-4883-adf2-452d0dd05b18-tls-certs" pod "55ce8a09-3219-4883-adf2-452d0dd05b18" (UID: "55ce8a09-3219-4883-adf2-452d0dd05b18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:24.456488 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.456466 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-dshm" pod "55ce8a09-3219-4883-adf2-452d0dd05b18" (UID: "55ce8a09-3219-4883-adf2-452d0dd05b18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:24.456896 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.456876 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ce8a09-3219-4883-adf2-452d0dd05b18-kube-api-access-gcz5q" pod "55ce8a09-3219-4883-adf2-452d0dd05b18" (UID: "55ce8a09-3219-4883-adf2-452d0dd05b18"). InnerVolumeSpecName "kube-api-access-gcz5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:24.501929 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.501901 5047 generic.go:356] "Generic (PLEG): container finished" podID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerID="3c5a3b406ee4b4e127256a19241ec5d9a97266a91c40e1e2b9aafc37a3689c6b" exitCode=0 Oct 30 20:04:24.502081 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.501947 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" Oct 30 20:04:24.502081 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.501981 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" event={"ID":"55ce8a09-3219-4883-adf2-452d0dd05b18","Type":"ContainerDied","Data":"3c5a3b406ee4b4e127256a19241ec5d9a97266a91c40e1e2b9aafc37a3689c6b"} Oct 30 20:04:24.502081 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.502052 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" event={"ID":"55ce8a09-3219-4883-adf2-452d0dd05b18","Type":"ContainerDied","Data":"06129f88a45de52bfa8adfa5d3e6895e15f7125c8a93c2f69e1215d1ae2fd58a"} Oct 30 20:04:24.502081 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.502079 5047 scope.go:122] "RemoveContainer" containerID="3c5a3b406ee4b4e127256a19241ec5d9a97266a91c40e1e2b9aafc37a3689c6b" Oct 30 20:04:24.503582 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.503557 5047 generic.go:356] "Generic (PLEG): container finished" podID="d6339b71-2746-47d8-810b-217e48b23e0e" containerID="714a7ead6ec214e8ddc5ec5e619959eb3c8d4fb4caf8562196648361dc820b89" exitCode=0 Oct 30 20:04:24.503738 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.503626 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" Oct 30 20:04:24.503738 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.503641 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" event={"ID":"d6339b71-2746-47d8-810b-217e48b23e0e","Type":"ContainerDied","Data":"714a7ead6ec214e8ddc5ec5e619959eb3c8d4fb4caf8562196648361dc820b89"} Oct 30 20:04:24.503738 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.503673 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4" event={"ID":"d6339b71-2746-47d8-810b-217e48b23e0e","Type":"ContainerDied","Data":"1a24d36b7fe5e635e5cc26fad29e066ee253ef5e61f95b9dec88b523844936ef"} Oct 30 20:04:24.514756 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.514673 5047 scope.go:122] "RemoveContainer" containerID="3c5a3b406ee4b4e127256a19241ec5d9a97266a91c40e1e2b9aafc37a3689c6b" Oct 30 20:04:24.515294 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:04:24.515261 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c5a3b406ee4b4e127256a19241ec5d9a97266a91c40e1e2b9aafc37a3689c6b\": container with ID starting with 3c5a3b406ee4b4e127256a19241ec5d9a97266a91c40e1e2b9aafc37a3689c6b not found: ID does not exist" containerID="3c5a3b406ee4b4e127256a19241ec5d9a97266a91c40e1e2b9aafc37a3689c6b" Oct 30 20:04:24.515402 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.515300 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5a3b406ee4b4e127256a19241ec5d9a97266a91c40e1e2b9aafc37a3689c6b"} err="failed to get container status \"3c5a3b406ee4b4e127256a19241ec5d9a97266a91c40e1e2b9aafc37a3689c6b\": rpc error: code = NotFound desc = could not find container \"3c5a3b406ee4b4e127256a19241ec5d9a97266a91c40e1e2b9aafc37a3689c6b\": container with ID starting with 3c5a3b406ee4b4e127256a19241ec5d9a97266a91c40e1e2b9aafc37a3689c6b not found: ID does not exist" Oct 30 20:04:24.515402 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.515320 5047 scope.go:122] "RemoveContainer" containerID="714a7ead6ec214e8ddc5ec5e619959eb3c8d4fb4caf8562196648361dc820b89" Oct 30 20:04:24.524078 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.524049 5047 scope.go:122] "RemoveContainer" containerID="714a7ead6ec214e8ddc5ec5e619959eb3c8d4fb4caf8562196648361dc820b89" Oct 30 20:04:24.524309 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:04:24.524291 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714a7ead6ec214e8ddc5ec5e619959eb3c8d4fb4caf8562196648361dc820b89\": container with ID starting with 714a7ead6ec214e8ddc5ec5e619959eb3c8d4fb4caf8562196648361dc820b89 not found: ID does not exist" containerID="714a7ead6ec214e8ddc5ec5e619959eb3c8d4fb4caf8562196648361dc820b89" Oct 30 20:04:24.524357 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.524315 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714a7ead6ec214e8ddc5ec5e619959eb3c8d4fb4caf8562196648361dc820b89"} err="failed to get container status \"714a7ead6ec214e8ddc5ec5e619959eb3c8d4fb4caf8562196648361dc820b89\": rpc error: code = NotFound desc = could not find container \"714a7ead6ec214e8ddc5ec5e619959eb3c8d4fb4caf8562196648361dc820b89\": container with ID starting with 714a7ead6ec214e8ddc5ec5e619959eb3c8d4fb4caf8562196648361dc820b89 not found: ID does not exist" Oct 30 20:04:24.553939 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.553915 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ce8a09-3219-4883-adf2-452d0dd05b18-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.554077 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.553944 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcz5q\" (UniqueName: \"kubernetes.io/projected/55ce8a09-3219-4883-adf2-452d0dd05b18-kube-api-access-gcz5q\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.554077 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.553958 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.554077 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.553971 5047 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6339b71-2746-47d8-810b-217e48b23e0e-tls-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.554077 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.553985 5047 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6339b71-2746-47d8-810b-217e48b23e0e-dshm\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.554077 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.553998 5047 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-model-cache\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.554077 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.554025 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mchkj\" (UniqueName: \"kubernetes.io/projected/d6339b71-2746-47d8-810b-217e48b23e0e-kube-api-access-mchkj\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.554077 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.554039 5047 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ce8a09-3219-4883-adf2-452d0dd05b18-tmp-dir\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:24.557945 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.557922 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6"] Oct 30 20:04:24.561760 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.561741 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6"] Oct 30 20:04:24.590314 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.590292 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4"] Oct 30 20:04:24.594714 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:24.594693 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e67b84/distinct-2-w5-e67b84-kserve-77545dcfdf-85jv4"] Oct 30 20:04:25.048366 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:25.048297 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" path="/var/lib/kubelet/pods/55ce8a09-3219-4883-adf2-452d0dd05b18/volumes" Oct 30 20:04:25.048710 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:25.048693 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" path="/var/lib/kubelet/pods/d6339b71-2746-47d8-810b-217e48b23e0e/volumes" Oct 30 20:04:25.334316 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:25.334272 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e67b84/distinct-w5-e67b84-kserve-54dc6f5bcd-vphp6" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 20:04:33.613225 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:33.613193 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm"] Oct 30 20:04:33.614105 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:33.614073 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" podUID="d0fa5318-4e5f-4970-ac2f-39a8e839b6a5" containerName="maas-api" containerID="cri-o://f8167ff50861b58c8a9acd647ba50fa8641cae0ee3767791bd4b48c3a420f646" gracePeriod=30 Oct 30 20:04:33.874592 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:33.874519 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 20:04:33.944633 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:33.944601 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-maas-api-tls\") pod \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\" (UID: \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\") " Oct 30 20:04:33.944808 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:33.944650 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-metrics-certs\") pod \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\" (UID: \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\") " Oct 30 20:04:33.944808 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:33.944676 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-kube-api-access-vxsck\" (UniqueName: \"kubernetes.io/projected/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-kube-api-access-vxsck\") pod \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\" (UID: \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\") " Oct 30 20:04:33.944808 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:33.944742 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-openshift-service-ca\") pod \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\" (UID: \"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5\") " Oct 30 20:04:33.945192 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:33.945164 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-openshift-service-ca" pod "d0fa5318-4e5f-4970-ac2f-39a8e839b6a5" (UID: "d0fa5318-4e5f-4970-ac2f-39a8e839b6a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:33.949478 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:33.949448 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-metrics-certs" pod "d0fa5318-4e5f-4970-ac2f-39a8e839b6a5" (UID: "d0fa5318-4e5f-4970-ac2f-39a8e839b6a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:33.949598 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:33.949504 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-maas-api-tls" pod "d0fa5318-4e5f-4970-ac2f-39a8e839b6a5" (UID: "d0fa5318-4e5f-4970-ac2f-39a8e839b6a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:33.949598 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:33.949515 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-kube-api-access-vxsck" pod "d0fa5318-4e5f-4970-ac2f-39a8e839b6a5" (UID: "d0fa5318-4e5f-4970-ac2f-39a8e839b6a5"). InnerVolumeSpecName "kube-api-access-vxsck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:34.045746 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:34.045713 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:34.045746 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:34.045742 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:34.045746 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:34.045751 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:34.045985 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:34.045760 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxsck\" (UniqueName: \"kubernetes.io/projected/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5-kube-api-access-vxsck\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:34.555980 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:34.555947 5047 generic.go:356] "Generic (PLEG): container finished" podID="d0fa5318-4e5f-4970-ac2f-39a8e839b6a5" containerID="f8167ff50861b58c8a9acd647ba50fa8641cae0ee3767791bd4b48c3a420f646" exitCode=0 Oct 30 20:04:34.556177 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:34.555992 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" event={"ID":"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5","Type":"ContainerDied","Data":"f8167ff50861b58c8a9acd647ba50fa8641cae0ee3767791bd4b48c3a420f646"} Oct 30 20:04:34.556177 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:34.555994 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" Oct 30 20:04:34.556177 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:34.556048 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm" event={"ID":"d0fa5318-4e5f-4970-ac2f-39a8e839b6a5","Type":"ContainerDied","Data":"2aaea9d86c98d83634fc5e2bb7c637560c4f53a9cab83d4d1fdf980659f6a531"} Oct 30 20:04:34.556177 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:34.556072 5047 scope.go:122] "RemoveContainer" containerID="f8167ff50861b58c8a9acd647ba50fa8641cae0ee3767791bd4b48c3a420f646" Oct 30 20:04:34.565619 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:34.565599 5047 scope.go:122] "RemoveContainer" containerID="f8167ff50861b58c8a9acd647ba50fa8641cae0ee3767791bd4b48c3a420f646" Oct 30 20:04:34.565883 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:04:34.565863 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8167ff50861b58c8a9acd647ba50fa8641cae0ee3767791bd4b48c3a420f646\": container with ID starting with f8167ff50861b58c8a9acd647ba50fa8641cae0ee3767791bd4b48c3a420f646 not found: ID does not exist" containerID="f8167ff50861b58c8a9acd647ba50fa8641cae0ee3767791bd4b48c3a420f646" Oct 30 20:04:34.565938 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:34.565893 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8167ff50861b58c8a9acd647ba50fa8641cae0ee3767791bd4b48c3a420f646"} err="failed to get container status \"f8167ff50861b58c8a9acd647ba50fa8641cae0ee3767791bd4b48c3a420f646\": rpc error: code = NotFound desc = could not find container \"f8167ff50861b58c8a9acd647ba50fa8641cae0ee3767791bd4b48c3a420f646\": container with ID starting with f8167ff50861b58c8a9acd647ba50fa8641cae0ee3767791bd4b48c3a420f646 not found: ID does not exist" Oct 30 20:04:34.600791 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:34.600749 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm"] Oct 30 20:04:34.610480 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:34.610454 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e67b84-5887bb8cf5-tw4bm"] Oct 30 20:04:35.047893 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:35.047856 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0fa5318-4e5f-4970-ac2f-39a8e839b6a5" path="/var/lib/kubelet/pods/d0fa5318-4e5f-4970-ac2f-39a8e839b6a5/volumes" Oct 30 20:04:39.599407 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:04:39.599372 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:04:41.155125 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:41.155091 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt"] Oct 30 20:04:41.155637 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:41.155346 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" podUID="98e541fb-4108-4b0a-8c6b-7e6587354f19" containerName="istio-proxy" containerID="cri-o://65c4ccb3168fb3d14dc1a87ad072693769ee49757b1868aa2777d51bdb1517c3" gracePeriod=30 Oct 30 20:04:45.968081 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:45.968041 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65cb454cc4-j4fbd"] Oct 30 20:04:45.968536 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:45.968265 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-65cb454cc4-j4fbd" podUID="201d1cd3-34eb-4a4e-85c2-7198c93942c0" containerName="manager" containerID="cri-o://38fa58e74c671b005d0965cb1100643d14834fb6402479057d0b41bb45028868" gracePeriod=30 Oct 30 20:04:46.284183 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.284155 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cb454cc4-j4fbd" Oct 30 20:04:46.417402 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.417368 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 20:04:46.464769 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.464719 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/201d1cd3-34eb-4a4e-85c2-7198c93942c0-webhook-cert\" (UniqueName: \"kubernetes.io/secret/201d1cd3-34eb-4a4e-85c2-7198c93942c0-webhook-cert\") pod \"201d1cd3-34eb-4a4e-85c2-7198c93942c0\" (UID: \"201d1cd3-34eb-4a4e-85c2-7198c93942c0\") " Oct 30 20:04:46.464969 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.464788 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/201d1cd3-34eb-4a4e-85c2-7198c93942c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/201d1cd3-34eb-4a4e-85c2-7198c93942c0-metrics-certs\") pod \"201d1cd3-34eb-4a4e-85c2-7198c93942c0\" (UID: \"201d1cd3-34eb-4a4e-85c2-7198c93942c0\") " Oct 30 20:04:46.464969 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.464868 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201d1cd3-34eb-4a4e-85c2-7198c93942c0-kube-api-access-6ffn6\" (UniqueName: \"kubernetes.io/projected/201d1cd3-34eb-4a4e-85c2-7198c93942c0-kube-api-access-6ffn6\") pod \"201d1cd3-34eb-4a4e-85c2-7198c93942c0\" (UID: \"201d1cd3-34eb-4a4e-85c2-7198c93942c0\") " Oct 30 20:04:46.466985 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.466954 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/201d1cd3-34eb-4a4e-85c2-7198c93942c0-webhook-cert" pod "201d1cd3-34eb-4a4e-85c2-7198c93942c0" (UID: "201d1cd3-34eb-4a4e-85c2-7198c93942c0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:46.467198 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.467175 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201d1cd3-34eb-4a4e-85c2-7198c93942c0-kube-api-access-6ffn6" pod "201d1cd3-34eb-4a4e-85c2-7198c93942c0" (UID: "201d1cd3-34eb-4a4e-85c2-7198c93942c0"). InnerVolumeSpecName "kube-api-access-6ffn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:46.467454 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.467437 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/201d1cd3-34eb-4a4e-85c2-7198c93942c0-metrics-certs" pod "201d1cd3-34eb-4a4e-85c2-7198c93942c0" (UID: "201d1cd3-34eb-4a4e-85c2-7198c93942c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:46.566263 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.566166 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-podinfo\") pod \"98e541fb-4108-4b0a-8c6b-7e6587354f19\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " Oct 30 20:04:46.566263 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.566248 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-envoy\") pod \"98e541fb-4108-4b0a-8c6b-7e6587354f19\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " Oct 30 20:04:46.566519 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.566293 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e541fb-4108-4b0a-8c6b-7e6587354f19-kube-api-access-fwd65\" (UniqueName: \"kubernetes.io/projected/98e541fb-4108-4b0a-8c6b-7e6587354f19-kube-api-access-fwd65\") pod \"98e541fb-4108-4b0a-8c6b-7e6587354f19\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " Oct 30 20:04:46.566519 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.566378 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e541fb-4108-4b0a-8c6b-7e6587354f19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e541fb-4108-4b0a-8c6b-7e6587354f19-istiod-ca-cert\") pod \"98e541fb-4108-4b0a-8c6b-7e6587354f19\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " Oct 30 20:04:46.566519 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.566414 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-credential-socket\") pod \"98e541fb-4108-4b0a-8c6b-7e6587354f19\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " Oct 30 20:04:46.566519 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.566461 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-token\" (UniqueName: \"kubernetes.io/projected/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-token\") pod \"98e541fb-4108-4b0a-8c6b-7e6587354f19\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " Oct 30 20:04:46.566750 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.566539 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-data\") pod \"98e541fb-4108-4b0a-8c6b-7e6587354f19\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " Oct 30 20:04:46.566750 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.566652 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-workload-certs\") pod \"98e541fb-4108-4b0a-8c6b-7e6587354f19\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " Oct 30 20:04:46.566750 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.566711 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-workload-socket\") pod \"98e541fb-4108-4b0a-8c6b-7e6587354f19\" (UID: \"98e541fb-4108-4b0a-8c6b-7e6587354f19\") " Oct 30 20:04:46.566912 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.566861 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-data" pod "98e541fb-4108-4b0a-8c6b-7e6587354f19" (UID: "98e541fb-4108-4b0a-8c6b-7e6587354f19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:46.567178 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.567125 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-workload-socket" pod "98e541fb-4108-4b0a-8c6b-7e6587354f19" (UID: "98e541fb-4108-4b0a-8c6b-7e6587354f19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:46.567178 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.567133 5047 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/201d1cd3-34eb-4a4e-85c2-7198c93942c0-webhook-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:46.567178 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.566871 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e541fb-4108-4b0a-8c6b-7e6587354f19-istiod-ca-cert" pod "98e541fb-4108-4b0a-8c6b-7e6587354f19" (UID: "98e541fb-4108-4b0a-8c6b-7e6587354f19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:46.567178 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.567164 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/201d1cd3-34eb-4a4e-85c2-7198c93942c0-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:46.567178 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.567175 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-workload-certs" pod "98e541fb-4108-4b0a-8c6b-7e6587354f19" (UID: "98e541fb-4108-4b0a-8c6b-7e6587354f19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:46.567595 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.567183 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ffn6\" (UniqueName: \"kubernetes.io/projected/201d1cd3-34eb-4a4e-85c2-7198c93942c0-kube-api-access-6ffn6\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:46.567595 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.567220 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:46.567595 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.567433 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-credential-socket" pod "98e541fb-4108-4b0a-8c6b-7e6587354f19" (UID: "98e541fb-4108-4b0a-8c6b-7e6587354f19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:46.568699 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.568677 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e541fb-4108-4b0a-8c6b-7e6587354f19-kube-api-access-fwd65" pod "98e541fb-4108-4b0a-8c6b-7e6587354f19" (UID: "98e541fb-4108-4b0a-8c6b-7e6587354f19"). InnerVolumeSpecName "kube-api-access-fwd65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:46.568825 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.568729 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-token" pod "98e541fb-4108-4b0a-8c6b-7e6587354f19" (UID: "98e541fb-4108-4b0a-8c6b-7e6587354f19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:46.568924 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.568906 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-podinfo" pod "98e541fb-4108-4b0a-8c6b-7e6587354f19" (UID: "98e541fb-4108-4b0a-8c6b-7e6587354f19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:46.568995 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.568910 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-envoy" pod "98e541fb-4108-4b0a-8c6b-7e6587354f19" (UID: "98e541fb-4108-4b0a-8c6b-7e6587354f19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:46.620545 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.620504 5047 generic.go:356] "Generic (PLEG): container finished" podID="201d1cd3-34eb-4a4e-85c2-7198c93942c0" containerID="38fa58e74c671b005d0965cb1100643d14834fb6402479057d0b41bb45028868" exitCode=0 Oct 30 20:04:46.620774 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.620559 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cb454cc4-j4fbd" Oct 30 20:04:46.620774 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.620591 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cb454cc4-j4fbd" event={"ID":"201d1cd3-34eb-4a4e-85c2-7198c93942c0","Type":"ContainerDied","Data":"38fa58e74c671b005d0965cb1100643d14834fb6402479057d0b41bb45028868"} Oct 30 20:04:46.620774 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.620650 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cb454cc4-j4fbd" event={"ID":"201d1cd3-34eb-4a4e-85c2-7198c93942c0","Type":"ContainerDied","Data":"d8c291318c3b781cb6000bdb9da6c3be6253ee15634b2d5a482c1045b506baee"} Oct 30 20:04:46.620774 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.620681 5047 scope.go:122] "RemoveContainer" containerID="38fa58e74c671b005d0965cb1100643d14834fb6402479057d0b41bb45028868" Oct 30 20:04:46.622148 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.622123 5047 generic.go:356] "Generic (PLEG): container finished" podID="98e541fb-4108-4b0a-8c6b-7e6587354f19" containerID="65c4ccb3168fb3d14dc1a87ad072693769ee49757b1868aa2777d51bdb1517c3" exitCode=0 Oct 30 20:04:46.622293 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.622173 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" event={"ID":"98e541fb-4108-4b0a-8c6b-7e6587354f19","Type":"ContainerDied","Data":"65c4ccb3168fb3d14dc1a87ad072693769ee49757b1868aa2777d51bdb1517c3"} Oct 30 20:04:46.622293 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.622180 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" Oct 30 20:04:46.622293 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.622203 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt" event={"ID":"98e541fb-4108-4b0a-8c6b-7e6587354f19","Type":"ContainerDied","Data":"d8777e08d1888c054ea87d132ca3012357bd1783a54c2c38aebe5cf9e569b230"} Oct 30 20:04:46.632767 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.632745 5047 scope.go:122] "RemoveContainer" containerID="38fa58e74c671b005d0965cb1100643d14834fb6402479057d0b41bb45028868" Oct 30 20:04:46.633091 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:04:46.633066 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38fa58e74c671b005d0965cb1100643d14834fb6402479057d0b41bb45028868\": container with ID starting with 38fa58e74c671b005d0965cb1100643d14834fb6402479057d0b41bb45028868 not found: ID does not exist" containerID="38fa58e74c671b005d0965cb1100643d14834fb6402479057d0b41bb45028868" Oct 30 20:04:46.633192 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.633104 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38fa58e74c671b005d0965cb1100643d14834fb6402479057d0b41bb45028868"} err="failed to get container status \"38fa58e74c671b005d0965cb1100643d14834fb6402479057d0b41bb45028868\": rpc error: code = NotFound desc = could not find container \"38fa58e74c671b005d0965cb1100643d14834fb6402479057d0b41bb45028868\": container with ID starting with 38fa58e74c671b005d0965cb1100643d14834fb6402479057d0b41bb45028868 not found: ID does not exist" Oct 30 20:04:46.633192 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.633131 5047 scope.go:122] "RemoveContainer" containerID="65c4ccb3168fb3d14dc1a87ad072693769ee49757b1868aa2777d51bdb1517c3" Oct 30 20:04:46.641693 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.641673 5047 scope.go:122] "RemoveContainer" containerID="65c4ccb3168fb3d14dc1a87ad072693769ee49757b1868aa2777d51bdb1517c3" Oct 30 20:04:46.641979 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:04:46.641962 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65c4ccb3168fb3d14dc1a87ad072693769ee49757b1868aa2777d51bdb1517c3\": container with ID starting with 65c4ccb3168fb3d14dc1a87ad072693769ee49757b1868aa2777d51bdb1517c3 not found: ID does not exist" containerID="65c4ccb3168fb3d14dc1a87ad072693769ee49757b1868aa2777d51bdb1517c3" Oct 30 20:04:46.642069 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.641986 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65c4ccb3168fb3d14dc1a87ad072693769ee49757b1868aa2777d51bdb1517c3"} err="failed to get container status \"65c4ccb3168fb3d14dc1a87ad072693769ee49757b1868aa2777d51bdb1517c3\": rpc error: code = NotFound desc = could not find container \"65c4ccb3168fb3d14dc1a87ad072693769ee49757b1868aa2777d51bdb1517c3\": container with ID starting with 65c4ccb3168fb3d14dc1a87ad072693769ee49757b1868aa2777d51bdb1517c3 not found: ID does not exist" Oct 30 20:04:46.666316 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.666280 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65cb454cc4-j4fbd"] Oct 30 20:04:46.668148 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.667788 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e541fb-4108-4b0a-8c6b-7e6587354f19-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:46.668148 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.667821 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:46.668148 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.667836 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:46.668148 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.667851 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:46.668148 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.667864 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:46.668148 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.667877 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:46.668148 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.667889 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e541fb-4108-4b0a-8c6b-7e6587354f19-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:46.668148 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.667902 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwd65\" (UniqueName: \"kubernetes.io/projected/98e541fb-4108-4b0a-8c6b-7e6587354f19-kube-api-access-fwd65\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:04:46.670643 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.670611 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65cb454cc4-j4fbd"] Oct 30 20:04:46.700560 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.700520 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt"] Oct 30 20:04:46.705410 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:46.705383 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e67b84-openshift-default-787fb8b88c-9v8bt"] Oct 30 20:04:47.051077 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:47.051038 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="201d1cd3-34eb-4a4e-85c2-7198c93942c0" path="/var/lib/kubelet/pods/201d1cd3-34eb-4a4e-85c2-7198c93942c0/volumes" Oct 30 20:04:47.051753 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:47.051730 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e541fb-4108-4b0a-8c6b-7e6587354f19" path="/var/lib/kubelet/pods/98e541fb-4108-4b0a-8c6b-7e6587354f19/volumes" Oct 30 20:04:48.068644 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.068561 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65cb454cc4-q7tpn"] Oct 30 20:04:48.069062 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.068989 5047 cpu_manager.go:414] "containerMap: removing container" podUID="201d1cd3-34eb-4a4e-85c2-7198c93942c0" containerName="manager" Oct 30 20:04:48.069062 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069001 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201d1cd3-34eb-4a4e-85c2-7198c93942c0" containerName="manager" Oct 30 20:04:48.069062 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069048 5047 cpu_manager.go:414] "containerMap: removing container" podUID="98e541fb-4108-4b0a-8c6b-7e6587354f19" containerName="istio-proxy" Oct 30 20:04:48.069062 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069053 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e541fb-4108-4b0a-8c6b-7e6587354f19" containerName="istio-proxy" Oct 30 20:04:48.069224 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069069 5047 cpu_manager.go:414] "containerMap: removing container" podUID="032d435a-1b1a-4bd4-acd6-968efd75afa4" containerName="revoke-keys" Oct 30 20:04:48.069224 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069076 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032d435a-1b1a-4bd4-acd6-968efd75afa4" containerName="revoke-keys" Oct 30 20:04:48.069224 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069090 5047 cpu_manager.go:414] "containerMap: removing container" podUID="d0fa5318-4e5f-4970-ac2f-39a8e839b6a5" containerName="maas-api" Oct 30 20:04:48.069224 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069095 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0fa5318-4e5f-4970-ac2f-39a8e839b6a5" containerName="maas-api" Oct 30 20:04:48.069224 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069104 5047 cpu_manager.go:414] "containerMap: removing container" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" Oct 30 20:04:48.069224 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069109 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" Oct 30 20:04:48.069224 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069120 5047 cpu_manager.go:414] "containerMap: removing container" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" Oct 30 20:04:48.069224 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069125 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" Oct 30 20:04:48.069224 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069189 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea906da9-9a7c-4755-b5f1-2f6388875be2" containerName="istio-proxy" Oct 30 20:04:48.069224 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069201 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="201d1cd3-34eb-4a4e-85c2-7198c93942c0" containerName="manager" Oct 30 20:04:48.069224 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069208 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e541fb-4108-4b0a-8c6b-7e6587354f19" containerName="istio-proxy" Oct 30 20:04:48.069224 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069216 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="032d435a-1b1a-4bd4-acd6-968efd75afa4" containerName="revoke-keys" Oct 30 20:04:48.069224 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069223 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0fa5318-4e5f-4970-ac2f-39a8e839b6a5" containerName="maas-api" Oct 30 20:04:48.069224 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069229 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6339b71-2746-47d8-810b-217e48b23e0e" containerName="main" Oct 30 20:04:48.069652 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.069236 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ce8a09-3219-4883-adf2-452d0dd05b18" containerName="main" Oct 30 20:04:48.073786 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.073766 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cb454cc4-q7tpn" Oct 30 20:04:48.091602 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.091574 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65cb454cc4-q7tpn"] Oct 30 20:04:48.099194 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.099139 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 20:04:48.099363 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.099193 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-52jw9\"" Oct 30 20:04:48.099363 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.099280 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 20:04:48.099363 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.099323 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 20:04:48.183292 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.183247 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b328855-9703-4253-bf09-1d88ba6c53d6-webhook-cert\") pod \"maas-controller-65cb454cc4-q7tpn\" (UID: \"7b328855-9703-4253-bf09-1d88ba6c53d6\") " pod="opendatahub/maas-controller-65cb454cc4-q7tpn" Oct 30 20:04:48.183481 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.183301 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g6q2\" (UniqueName: \"kubernetes.io/projected/7b328855-9703-4253-bf09-1d88ba6c53d6-kube-api-access-4g6q2\") pod \"maas-controller-65cb454cc4-q7tpn\" (UID: \"7b328855-9703-4253-bf09-1d88ba6c53d6\") " pod="opendatahub/maas-controller-65cb454cc4-q7tpn" Oct 30 20:04:48.183481 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.183448 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b328855-9703-4253-bf09-1d88ba6c53d6-metrics-certs\") pod \"maas-controller-65cb454cc4-q7tpn\" (UID: \"7b328855-9703-4253-bf09-1d88ba6c53d6\") " pod="opendatahub/maas-controller-65cb454cc4-q7tpn" Oct 30 20:04:48.284654 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.284619 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b328855-9703-4253-bf09-1d88ba6c53d6-metrics-certs\") pod \"maas-controller-65cb454cc4-q7tpn\" (UID: \"7b328855-9703-4253-bf09-1d88ba6c53d6\") " pod="opendatahub/maas-controller-65cb454cc4-q7tpn" Oct 30 20:04:48.284879 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.284687 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b328855-9703-4253-bf09-1d88ba6c53d6-webhook-cert\") pod \"maas-controller-65cb454cc4-q7tpn\" (UID: \"7b328855-9703-4253-bf09-1d88ba6c53d6\") " pod="opendatahub/maas-controller-65cb454cc4-q7tpn" Oct 30 20:04:48.284879 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.284712 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g6q2\" (UniqueName: \"kubernetes.io/projected/7b328855-9703-4253-bf09-1d88ba6c53d6-kube-api-access-4g6q2\") pod \"maas-controller-65cb454cc4-q7tpn\" (UID: \"7b328855-9703-4253-bf09-1d88ba6c53d6\") " pod="opendatahub/maas-controller-65cb454cc4-q7tpn" Oct 30 20:04:48.287266 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.287231 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b328855-9703-4253-bf09-1d88ba6c53d6-webhook-cert\") pod \"maas-controller-65cb454cc4-q7tpn\" (UID: \"7b328855-9703-4253-bf09-1d88ba6c53d6\") " pod="opendatahub/maas-controller-65cb454cc4-q7tpn" Oct 30 20:04:48.287381 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.287270 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b328855-9703-4253-bf09-1d88ba6c53d6-metrics-certs\") pod \"maas-controller-65cb454cc4-q7tpn\" (UID: \"7b328855-9703-4253-bf09-1d88ba6c53d6\") " pod="opendatahub/maas-controller-65cb454cc4-q7tpn" Oct 30 20:04:48.301331 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.301301 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g6q2\" (UniqueName: \"kubernetes.io/projected/7b328855-9703-4253-bf09-1d88ba6c53d6-kube-api-access-4g6q2\") pod \"maas-controller-65cb454cc4-q7tpn\" (UID: \"7b328855-9703-4253-bf09-1d88ba6c53d6\") " pod="opendatahub/maas-controller-65cb454cc4-q7tpn" Oct 30 20:04:48.389065 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.389020 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cb454cc4-q7tpn" Oct 30 20:04:48.683478 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:48.683451 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65cb454cc4-q7tpn"] Oct 30 20:04:48.685503 ip-10-0-140-235 kubenswrapper[5047]: W1030 20:04:48.685469 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b328855_9703_4253_bf09_1d88ba6c53d6.slice/crio-0000d7106b76e3c0f88725e90d0083a6b2ec7b44c077928d756d7af857a7df65 WatchSource:0}: Error finding container 0000d7106b76e3c0f88725e90d0083a6b2ec7b44c077928d756d7af857a7df65: Status 404 returned error can't find the container with id 0000d7106b76e3c0f88725e90d0083a6b2ec7b44c077928d756d7af857a7df65 Oct 30 20:04:49.641051 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:49.640982 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cb454cc4-q7tpn" event={"ID":"7b328855-9703-4253-bf09-1d88ba6c53d6","Type":"ContainerStarted","Data":"3cf92dffcbb84f68405c2939f9bdaa7fa399cb7839379665624c41092fdf9ce9"} Oct 30 20:04:49.641477 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:49.641077 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cb454cc4-q7tpn" event={"ID":"7b328855-9703-4253-bf09-1d88ba6c53d6","Type":"ContainerStarted","Data":"0000d7106b76e3c0f88725e90d0083a6b2ec7b44c077928d756d7af857a7df65"} Oct 30 20:04:49.641477 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:49.641108 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65cb454cc4-q7tpn" Oct 30 20:04:49.676513 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:49.676459 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-65cb454cc4-q7tpn" podStartSLOduration=1.2554246949999999 podStartE2EDuration="1.676444895s" totalImagesPullingTime="421.0202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:49.675207134 +0000 UTC m=+3642.749662493" watchObservedRunningTime="2026-10-30 20:04:49.676444895 +0000 UTC m=+3642.750900255" Oct 30 20:04:55.570860 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.570816 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb"] Oct 30 20:04:55.574513 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.574490 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.605614 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.605578 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb"] Oct 30 20:04:55.648423 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.648393 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.648627 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.648440 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.648627 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.648459 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.648627 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.648573 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.648747 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.648640 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.648747 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.648695 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.648747 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.648717 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.648747 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.648740 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.648889 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.648754 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwdfj\" (UniqueName: \"kubernetes.io/projected/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-kube-api-access-hwdfj\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.749441 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.749403 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.749441 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.749439 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.749668 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.749456 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.749668 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.749501 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.749668 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.749555 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.749668 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.749580 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.749668 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.749616 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.749668 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.749638 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwdfj\" (UniqueName: \"kubernetes.io/projected/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-kube-api-access-hwdfj\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.749918 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.749691 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.749918 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.749883 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.750195 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.750168 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.750359 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.750216 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.750359 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.750310 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.750359 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.750342 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.752159 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.752130 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.752338 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.752319 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.767168 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.767140 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.767260 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.767240 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwdfj\" (UniqueName: \"kubernetes.io/projected/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-kube-api-access-hwdfj\") pod \"maas-default-gateway-openshift-default-6f54779779-mpgxb\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:55.888167 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:55.888130 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:56.192135 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:56.192106 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb"] Oct 30 20:04:56.194153 ip-10-0-140-235 kubenswrapper[5047]: W1030 20:04:56.194126 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbda9b5f_24c3_4cbc_ba7f_d134eda7adce.slice/crio-d35ec7f6c135167cab395cc77d74637152edcf947e48b01d4c458410ce95cf92 WatchSource:0}: Error finding container d35ec7f6c135167cab395cc77d74637152edcf947e48b01d4c458410ce95cf92: Status 404 returned error can't find the container with id d35ec7f6c135167cab395cc77d74637152edcf947e48b01d4c458410ce95cf92 Oct 30 20:04:56.196249 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:56.196214 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:56.196323 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:56.196285 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:56.196323 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:56.196311 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:56.676304 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:56.676265 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" event={"ID":"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce","Type":"ContainerStarted","Data":"0f2a8b23795e8ef06cf2df563cc29498c02ad9a295d5c3c781afcd086e4bc8c8"} Oct 30 20:04:56.676857 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:56.676325 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" event={"ID":"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce","Type":"ContainerStarted","Data":"d35ec7f6c135167cab395cc77d74637152edcf947e48b01d4c458410ce95cf92"} Oct 30 20:04:56.733859 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:56.733792 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" podStartSLOduration=1.733777131 podStartE2EDuration="1.733777131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:56.722986175 +0000 UTC m=+3649.797441534" watchObservedRunningTime="2026-10-30 20:04:56.733777131 +0000 UTC m=+3649.808232489" Oct 30 20:04:56.889125 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:56.889087 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:04:57.890049 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:57.889986 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-mpgxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 20:04:57.890447 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:57.890066 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" podUID="dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 20:04:58.888927 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:58.888888 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-mpgxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 20:04:58.889153 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:58.888957 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" podUID="dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 20:04:59.888773 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:59.888733 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-mpgxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 20:04:59.889207 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:04:59.888791 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" podUID="dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 20:05:00.660586 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:00.660543 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65cb454cc4-q7tpn" Oct 30 20:05:00.888529 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:00.888493 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-mpgxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 20:05:00.888743 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:00.888546 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" podUID="dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 20:05:01.888537 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:01.888497 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-mpgxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 20:05:01.888977 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:01.888570 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" podUID="dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 20:05:02.889035 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:02.888985 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-mpgxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 20:05:02.889413 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:02.889058 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" podUID="dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 20:05:04.338576 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:04.338528 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:05:04.339137 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:04.338920 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:05:04.339796 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:04.339773 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:05:09.600370 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:05:09.600331 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:05:20.303741 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.303698 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65cb454cc4-q7tpn"] Oct 30 20:05:20.304274 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.303929 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-65cb454cc4-q7tpn" podUID="7b328855-9703-4253-bf09-1d88ba6c53d6" containerName="manager" containerID="cri-o://3cf92dffcbb84f68405c2939f9bdaa7fa399cb7839379665624c41092fdf9ce9" gracePeriod=30 Oct 30 20:05:20.568812 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.568747 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cb454cc4-q7tpn" Oct 30 20:05:20.695071 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.695038 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b328855-9703-4253-bf09-1d88ba6c53d6-kube-api-access-4g6q2\" (UniqueName: \"kubernetes.io/projected/7b328855-9703-4253-bf09-1d88ba6c53d6-kube-api-access-4g6q2\") pod \"7b328855-9703-4253-bf09-1d88ba6c53d6\" (UID: \"7b328855-9703-4253-bf09-1d88ba6c53d6\") " Oct 30 20:05:20.695283 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.695105 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b328855-9703-4253-bf09-1d88ba6c53d6-webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b328855-9703-4253-bf09-1d88ba6c53d6-webhook-cert\") pod \"7b328855-9703-4253-bf09-1d88ba6c53d6\" (UID: \"7b328855-9703-4253-bf09-1d88ba6c53d6\") " Oct 30 20:05:20.695283 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.695177 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b328855-9703-4253-bf09-1d88ba6c53d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b328855-9703-4253-bf09-1d88ba6c53d6-metrics-certs\") pod \"7b328855-9703-4253-bf09-1d88ba6c53d6\" (UID: \"7b328855-9703-4253-bf09-1d88ba6c53d6\") " Oct 30 20:05:20.697183 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.697148 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b328855-9703-4253-bf09-1d88ba6c53d6-webhook-cert" pod "7b328855-9703-4253-bf09-1d88ba6c53d6" (UID: "7b328855-9703-4253-bf09-1d88ba6c53d6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:05:20.697316 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.697186 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b328855-9703-4253-bf09-1d88ba6c53d6-kube-api-access-4g6q2" pod "7b328855-9703-4253-bf09-1d88ba6c53d6" (UID: "7b328855-9703-4253-bf09-1d88ba6c53d6"). InnerVolumeSpecName "kube-api-access-4g6q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:20.697316 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.697203 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b328855-9703-4253-bf09-1d88ba6c53d6-metrics-certs" pod "7b328855-9703-4253-bf09-1d88ba6c53d6" (UID: "7b328855-9703-4253-bf09-1d88ba6c53d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:05:20.796687 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.796644 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g6q2\" (UniqueName: \"kubernetes.io/projected/7b328855-9703-4253-bf09-1d88ba6c53d6-kube-api-access-4g6q2\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:05:20.796687 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.796681 5047 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b328855-9703-4253-bf09-1d88ba6c53d6-webhook-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:05:20.796687 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.796693 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b328855-9703-4253-bf09-1d88ba6c53d6-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:05:20.805079 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.805046 5047 generic.go:356] "Generic (PLEG): container finished" podID="7b328855-9703-4253-bf09-1d88ba6c53d6" containerID="3cf92dffcbb84f68405c2939f9bdaa7fa399cb7839379665624c41092fdf9ce9" exitCode=0 Oct 30 20:05:20.805269 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.805094 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cb454cc4-q7tpn" Oct 30 20:05:20.805269 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.805117 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cb454cc4-q7tpn" event={"ID":"7b328855-9703-4253-bf09-1d88ba6c53d6","Type":"ContainerDied","Data":"3cf92dffcbb84f68405c2939f9bdaa7fa399cb7839379665624c41092fdf9ce9"} Oct 30 20:05:20.805269 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.805214 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cb454cc4-q7tpn" event={"ID":"7b328855-9703-4253-bf09-1d88ba6c53d6","Type":"ContainerDied","Data":"0000d7106b76e3c0f88725e90d0083a6b2ec7b44c077928d756d7af857a7df65"} Oct 30 20:05:20.805269 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.805237 5047 scope.go:122] "RemoveContainer" containerID="3cf92dffcbb84f68405c2939f9bdaa7fa399cb7839379665624c41092fdf9ce9" Oct 30 20:05:20.815795 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.815770 5047 scope.go:122] "RemoveContainer" containerID="3cf92dffcbb84f68405c2939f9bdaa7fa399cb7839379665624c41092fdf9ce9" Oct 30 20:05:20.816123 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:05:20.816100 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf92dffcbb84f68405c2939f9bdaa7fa399cb7839379665624c41092fdf9ce9\": container with ID starting with 3cf92dffcbb84f68405c2939f9bdaa7fa399cb7839379665624c41092fdf9ce9 not found: ID does not exist" containerID="3cf92dffcbb84f68405c2939f9bdaa7fa399cb7839379665624c41092fdf9ce9" Oct 30 20:05:20.816204 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.816136 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf92dffcbb84f68405c2939f9bdaa7fa399cb7839379665624c41092fdf9ce9"} err="failed to get container status \"3cf92dffcbb84f68405c2939f9bdaa7fa399cb7839379665624c41092fdf9ce9\": rpc error: code = NotFound desc = could not find container \"3cf92dffcbb84f68405c2939f9bdaa7fa399cb7839379665624c41092fdf9ce9\": container with ID starting with 3cf92dffcbb84f68405c2939f9bdaa7fa399cb7839379665624c41092fdf9ce9 not found: ID does not exist" Oct 30 20:05:20.850025 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.849969 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65cb454cc4-q7tpn"] Oct 30 20:05:20.853585 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:20.853549 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65cb454cc4-q7tpn"] Oct 30 20:05:21.047775 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:21.047737 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b328855-9703-4253-bf09-1d88ba6c53d6" path="/var/lib/kubelet/pods/7b328855-9703-4253-bf09-1d88ba6c53d6/volumes" Oct 30 20:05:22.045846 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.045814 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65cb454cc4-5lnl7"] Oct 30 20:05:22.046253 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.046243 5047 cpu_manager.go:414] "containerMap: removing container" podUID="7b328855-9703-4253-bf09-1d88ba6c53d6" containerName="manager" Oct 30 20:05:22.046302 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.046255 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b328855-9703-4253-bf09-1d88ba6c53d6" containerName="manager" Oct 30 20:05:22.046341 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.046334 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b328855-9703-4253-bf09-1d88ba6c53d6" containerName="manager" Oct 30 20:05:22.050813 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.050789 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cb454cc4-5lnl7" Oct 30 20:05:22.063412 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.063392 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 20:05:22.073571 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.073521 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 20:05:22.073725 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.073523 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-52jw9\"" Oct 30 20:05:22.073725 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.073587 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 20:05:22.077891 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.077869 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65cb454cc4-5lnl7"] Oct 30 20:05:22.109926 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.109899 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w49h5\" (UniqueName: \"kubernetes.io/projected/70ddcddf-b107-4a4a-ac56-4b451ead2612-kube-api-access-w49h5\") pod \"maas-controller-65cb454cc4-5lnl7\" (UID: \"70ddcddf-b107-4a4a-ac56-4b451ead2612\") " pod="opendatahub/maas-controller-65cb454cc4-5lnl7" Oct 30 20:05:22.110077 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.109978 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ddcddf-b107-4a4a-ac56-4b451ead2612-metrics-certs\") pod \"maas-controller-65cb454cc4-5lnl7\" (UID: \"70ddcddf-b107-4a4a-ac56-4b451ead2612\") " pod="opendatahub/maas-controller-65cb454cc4-5lnl7" Oct 30 20:05:22.110077 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.110039 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ddcddf-b107-4a4a-ac56-4b451ead2612-webhook-cert\") pod \"maas-controller-65cb454cc4-5lnl7\" (UID: \"70ddcddf-b107-4a4a-ac56-4b451ead2612\") " pod="opendatahub/maas-controller-65cb454cc4-5lnl7" Oct 30 20:05:22.210735 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.210702 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w49h5\" (UniqueName: \"kubernetes.io/projected/70ddcddf-b107-4a4a-ac56-4b451ead2612-kube-api-access-w49h5\") pod \"maas-controller-65cb454cc4-5lnl7\" (UID: \"70ddcddf-b107-4a4a-ac56-4b451ead2612\") " pod="opendatahub/maas-controller-65cb454cc4-5lnl7" Oct 30 20:05:22.210936 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.210766 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ddcddf-b107-4a4a-ac56-4b451ead2612-metrics-certs\") pod \"maas-controller-65cb454cc4-5lnl7\" (UID: \"70ddcddf-b107-4a4a-ac56-4b451ead2612\") " pod="opendatahub/maas-controller-65cb454cc4-5lnl7" Oct 30 20:05:22.210936 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.210795 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ddcddf-b107-4a4a-ac56-4b451ead2612-webhook-cert\") pod \"maas-controller-65cb454cc4-5lnl7\" (UID: \"70ddcddf-b107-4a4a-ac56-4b451ead2612\") " pod="opendatahub/maas-controller-65cb454cc4-5lnl7" Oct 30 20:05:22.213361 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.213339 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ddcddf-b107-4a4a-ac56-4b451ead2612-webhook-cert\") pod \"maas-controller-65cb454cc4-5lnl7\" (UID: \"70ddcddf-b107-4a4a-ac56-4b451ead2612\") " pod="opendatahub/maas-controller-65cb454cc4-5lnl7" Oct 30 20:05:22.213432 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.213401 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ddcddf-b107-4a4a-ac56-4b451ead2612-metrics-certs\") pod \"maas-controller-65cb454cc4-5lnl7\" (UID: \"70ddcddf-b107-4a4a-ac56-4b451ead2612\") " pod="opendatahub/maas-controller-65cb454cc4-5lnl7" Oct 30 20:05:22.228594 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.228552 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w49h5\" (UniqueName: \"kubernetes.io/projected/70ddcddf-b107-4a4a-ac56-4b451ead2612-kube-api-access-w49h5\") pod \"maas-controller-65cb454cc4-5lnl7\" (UID: \"70ddcddf-b107-4a4a-ac56-4b451ead2612\") " pod="opendatahub/maas-controller-65cb454cc4-5lnl7" Oct 30 20:05:22.364000 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.363972 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cb454cc4-5lnl7" Oct 30 20:05:22.656280 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.656250 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65cb454cc4-5lnl7"] Oct 30 20:05:22.658653 ip-10-0-140-235 kubenswrapper[5047]: W1030 20:05:22.658621 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ddcddf_b107_4a4a_ac56_4b451ead2612.slice/crio-201d51761ccbbd316c1525b28b01d8fae0797d1309b55b537d065e7c5efc9d82 WatchSource:0}: Error finding container 201d51761ccbbd316c1525b28b01d8fae0797d1309b55b537d065e7c5efc9d82: Status 404 returned error can't find the container with id 201d51761ccbbd316c1525b28b01d8fae0797d1309b55b537d065e7c5efc9d82 Oct 30 20:05:22.817059 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:22.817025 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cb454cc4-5lnl7" event={"ID":"70ddcddf-b107-4a4a-ac56-4b451ead2612","Type":"ContainerStarted","Data":"201d51761ccbbd316c1525b28b01d8fae0797d1309b55b537d065e7c5efc9d82"} Oct 30 20:05:23.823813 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:23.823774 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cb454cc4-5lnl7" event={"ID":"70ddcddf-b107-4a4a-ac56-4b451ead2612","Type":"ContainerStarted","Data":"0762e49386168967bf4200cb45fc2485b4628df0858e3b39eb0f6d93064a4574"} Oct 30 20:05:23.824281 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:23.823913 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65cb454cc4-5lnl7" Oct 30 20:05:23.860653 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:23.860600 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-65cb454cc4-5lnl7" podStartSLOduration=1.542935486 podStartE2EDuration="1.860585715s" totalImagesPullingTime="317.650229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:23.860026988 +0000 UTC m=+3676.934482340" watchObservedRunningTime="2026-10-30 20:05:23.860585715 +0000 UTC m=+3676.935041071" Oct 30 20:05:34.831840 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:05:34.831806 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65cb454cc4-5lnl7" Oct 30 20:05:39.600593 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:05:39.600549 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:06:09.601403 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:06:09.601353 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:06:10.835553 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.835509 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4"] Oct 30 20:06:10.840019 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.839981 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:10.850258 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.850227 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk"] Oct 30 20:06:10.855238 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.855213 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:10.873065 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.873031 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4"] Oct 30 20:06:10.878849 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.878813 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk"] Oct 30 20:06:10.997140 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997097 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:10.997336 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997142 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:10.997336 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997222 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:10.997336 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997254 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:10.997336 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997289 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:10.997336 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997316 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:10.997538 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997438 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:10.997538 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997479 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:10.997538 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997512 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:10.997646 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997568 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:10.997646 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997605 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s47b8\" (UniqueName: \"kubernetes.io/projected/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-kube-api-access-s47b8\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:10.997714 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997665 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:10.997714 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997685 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:10.997714 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997705 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:10.997815 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997729 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mht8\" (UniqueName: \"kubernetes.io/projected/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-kube-api-access-6mht8\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:10.997815 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997759 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:10.997882 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997815 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:10.997882 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:10.997856 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.099404 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099298 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.099404 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099344 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.099404 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099370 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.099737 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099407 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mht8\" (UniqueName: \"kubernetes.io/projected/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-kube-api-access-6mht8\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.099737 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099433 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.099737 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099467 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.099737 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099508 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.099737 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099533 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.099737 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099553 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.099737 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099578 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.099737 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099604 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.099737 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099648 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.099737 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099676 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.099737 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099716 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.100391 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099756 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.100391 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099786 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.100391 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099836 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.100391 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099870 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s47b8\" (UniqueName: \"kubernetes.io/projected/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-kube-api-access-s47b8\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.100391 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099875 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.100391 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099909 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.100391 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.099948 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.100391 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.100193 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.100859 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.100615 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.100859 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.100810 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.100960 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.100930 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.101230 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.101076 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.101230 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.101115 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.101230 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.101183 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.102583 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.102550 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.102693 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.102550 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.102693 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.102618 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.102983 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.102967 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.138582 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.138539 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.138752 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.138683 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mht8\" (UniqueName: \"kubernetes.io/projected/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-kube-api-access-6mht8\") pod \"maas-default-gateway-openshift-default-6f54779779-xljwk\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.142124 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.142096 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.143976 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.143945 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s47b8\" (UniqueName: \"kubernetes.io/projected/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-kube-api-access-s47b8\") pod \"maas-default-gateway-openshift-default-6f54779779-bb2r4\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.154632 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.154601 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:11.169233 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.169195 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:11.494039 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.493997 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4"] Oct 30 20:06:11.496960 ip-10-0-140-235 kubenswrapper[5047]: W1030 20:06:11.496930 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2834fba3_6ed1_4b0c_ad61_bf9e41da5bbb.slice/crio-be335385a0f8f9868ef6243a43cacb05e225e2e7b74c70046928bcd1088f3503 WatchSource:0}: Error finding container be335385a0f8f9868ef6243a43cacb05e225e2e7b74c70046928bcd1088f3503: Status 404 returned error can't find the container with id be335385a0f8f9868ef6243a43cacb05e225e2e7b74c70046928bcd1088f3503 Oct 30 20:06:11.499281 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.499245 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:11.499373 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.499315 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:11.499373 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.499351 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:11.520918 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.520888 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk"] Oct 30 20:06:11.529212 ip-10-0-140-235 kubenswrapper[5047]: W1030 20:06:11.529186 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5671acbc_24a4_46dd_aa3f_aa7a25b6c8d4.slice/crio-fac33d4f408f89a3e2207a13e026d29d253a071866bb4c499ce0b553d887f826 WatchSource:0}: Error finding container fac33d4f408f89a3e2207a13e026d29d253a071866bb4c499ce0b553d887f826: Status 404 returned error can't find the container with id fac33d4f408f89a3e2207a13e026d29d253a071866bb4c499ce0b553d887f826 Oct 30 20:06:11.531725 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.531690 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:11.531804 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.531764 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:11.531844 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.531811 5047 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:11.606515 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:11.606479 5047 scope.go:122] "RemoveContainer" containerID="2ccd7acd3c6787922296acfdf072bf33e68ec34f2d58c294ba41cd7a6de14637" Oct 30 20:06:12.077493 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:12.077459 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" event={"ID":"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4","Type":"ContainerStarted","Data":"cc0f72e74a64faad186056bbd980546c6e6021bb54d755d1d0fe52b13f4a52a5"} Oct 30 20:06:12.077990 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:12.077969 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" event={"ID":"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4","Type":"ContainerStarted","Data":"fac33d4f408f89a3e2207a13e026d29d253a071866bb4c499ce0b553d887f826"} Oct 30 20:06:12.081438 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:12.081383 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" event={"ID":"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb","Type":"ContainerStarted","Data":"7cd34f89f7de6a092610ea8f6db53a6b8875a25a94e797a0f519993e4bb3a63a"} Oct 30 20:06:12.081686 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:12.081449 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" event={"ID":"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb","Type":"ContainerStarted","Data":"be335385a0f8f9868ef6243a43cacb05e225e2e7b74c70046928bcd1088f3503"} Oct 30 20:06:12.125852 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:12.125792 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" podStartSLOduration=2.125776593 podStartE2EDuration="2.125776593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:12.124862747 +0000 UTC m=+3725.199318106" watchObservedRunningTime="2026-10-30 20:06:12.125776593 +0000 UTC m=+3725.200231951" Oct 30 20:06:12.170029 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:12.162428 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:12.176032 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:12.175157 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 20:06:12.176032 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:12.175231 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 20:06:12.178033 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:12.177436 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:12.178033 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:12.177605 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" podStartSLOduration=2.177586427 podStartE2EDuration="2.177586427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:12.175943378 +0000 UTC m=+3725.250398743" watchObservedRunningTime="2026-10-30 20:06:12.177586427 +0000 UTC m=+3725.252041786" Oct 30 20:06:12.183032 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:12.181634 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xljwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 20:06:12.183032 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:12.181697 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 20:06:13.166497 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:13.166452 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 20:06:13.167131 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:13.167097 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 20:06:13.171467 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:13.171430 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xljwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 20:06:13.171717 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:13.171688 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 20:06:14.156030 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:14.155516 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 20:06:14.156030 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:14.155602 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 20:06:14.176997 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:14.176958 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xljwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 20:06:14.177641 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:14.177611 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 20:06:15.158770 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:15.158726 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 20:06:15.159133 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:15.159098 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 20:06:15.175580 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:15.175536 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xljwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 20:06:15.175892 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:15.175857 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 20:06:15.633154 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:15.633111 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-frwdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:15.633828 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:15.633792 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" podUID="5fcb783a-2210-41a3-8379-b2c417ac94d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:16.164029 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:16.163573 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 20:06:16.164029 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:16.163682 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 20:06:16.178186 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:16.178140 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xljwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 20:06:16.178484 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:16.178451 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 20:06:17.156641 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:17.156587 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 20:06:17.157335 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:17.157292 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 20:06:17.180555 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:17.180509 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xljwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 20:06:17.180886 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:17.180847 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 20:06:18.156256 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:18.156166 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 20:06:18.156256 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:18.156231 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 20:06:18.170805 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:18.170761 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xljwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 20:06:18.171201 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:18.170834 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 20:06:19.155716 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:19.155677 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 20:06:19.155892 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:19.155742 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 20:06:19.170167 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:19.170125 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xljwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 20:06:19.170335 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:19.170186 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 20:06:21.196256 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:21.196209 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:21.196893 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:21.196856 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:21.211162 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:21.211128 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xljwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:21.211302 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:21.211188 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:22.239214 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:22.239166 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:22.239765 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:22.239258 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:22.252312 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:22.252255 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xljwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:22.252627 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:22.252559 5047 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:22.630195 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:22.630156 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:22.630724 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:22.630665 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:22.631611 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:22.631588 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:06:22.676888 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:22.676852 5047 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:22.677253 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:22.677234 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:22.677950 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:06:22.677933 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:06:39.601843 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:06:39.601801 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:07:09.602522 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:07:09.602480 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:07:39.601532 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:07:39.601485 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:08:09.605051 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:08:09.604996 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:08:39.744331 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:08:39.744271 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:09:08.028667 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:09:08.028635 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 20:09:08.034120 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:09:08.034095 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 20:09:09.754283 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:09:09.754234 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:09:21.027223 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:09:21.027126 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn"] Oct 30 20:09:21.038776 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:09:21.038737 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-168c11f5-2b6mn"] Oct 30 20:09:21.049189 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:09:21.049154 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="032d435a-1b1a-4bd4-acd6-968efd75afa4" path="/var/lib/kubelet/pods/032d435a-1b1a-4bd4-acd6-968efd75afa4/volumes" Oct 30 20:09:39.601705 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:09:39.601661 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:10:09.601316 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:10:09.601272 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:10:19.589254 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:19.589218 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz"] Oct 30 20:10:19.593358 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:19.593332 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:10:19.625347 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:19.625289 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-013647-serving-cert\"" Oct 30 20:10:19.628762 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:19.628724 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz"] Oct 30 20:10:19.662465 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:19.662426 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/921657da-0e67-403c-8841-7dcaf528268c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-013647-68b55684db-4tglz\" (UID: \"921657da-0e67-403c-8841-7dcaf528268c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:10:19.662649 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:19.662476 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/921657da-0e67-403c-8841-7dcaf528268c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-013647-68b55684db-4tglz\" (UID: \"921657da-0e67-403c-8841-7dcaf528268c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:10:19.662649 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:19.662589 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/921657da-0e67-403c-8841-7dcaf528268c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-013647-68b55684db-4tglz\" (UID: \"921657da-0e67-403c-8841-7dcaf528268c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:10:19.662734 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:19.662654 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkf8d\" (UniqueName: \"kubernetes.io/projected/921657da-0e67-403c-8841-7dcaf528268c-kube-api-access-jkf8d\") pod \"maas-api-e2e-worker-single-main-013647-68b55684db-4tglz\" (UID: \"921657da-0e67-403c-8841-7dcaf528268c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:10:19.763585 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:19.763549 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkf8d\" (UniqueName: \"kubernetes.io/projected/921657da-0e67-403c-8841-7dcaf528268c-kube-api-access-jkf8d\") pod \"maas-api-e2e-worker-single-main-013647-68b55684db-4tglz\" (UID: \"921657da-0e67-403c-8841-7dcaf528268c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:10:19.763793 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:19.763606 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/921657da-0e67-403c-8841-7dcaf528268c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-013647-68b55684db-4tglz\" (UID: \"921657da-0e67-403c-8841-7dcaf528268c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:10:19.763793 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:19.763717 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/921657da-0e67-403c-8841-7dcaf528268c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-013647-68b55684db-4tglz\" (UID: \"921657da-0e67-403c-8841-7dcaf528268c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:10:19.763793 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:19.763759 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/921657da-0e67-403c-8841-7dcaf528268c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-013647-68b55684db-4tglz\" (UID: \"921657da-0e67-403c-8841-7dcaf528268c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:10:19.764506 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:19.764481 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/921657da-0e67-403c-8841-7dcaf528268c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-013647-68b55684db-4tglz\" (UID: \"921657da-0e67-403c-8841-7dcaf528268c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:10:19.766430 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:19.766399 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/921657da-0e67-403c-8841-7dcaf528268c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-013647-68b55684db-4tglz\" (UID: \"921657da-0e67-403c-8841-7dcaf528268c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:10:19.766538 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:19.766399 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/921657da-0e67-403c-8841-7dcaf528268c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-013647-68b55684db-4tglz\" (UID: \"921657da-0e67-403c-8841-7dcaf528268c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:10:19.782775 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:19.782740 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkf8d\" (UniqueName: \"kubernetes.io/projected/921657da-0e67-403c-8841-7dcaf528268c-kube-api-access-jkf8d\") pod \"maas-api-e2e-worker-single-main-013647-68b55684db-4tglz\" (UID: \"921657da-0e67-403c-8841-7dcaf528268c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:10:19.906404 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:19.906318 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:10:20.234592 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:20.234502 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz"] Oct 30 20:10:20.238093 ip-10-0-140-235 kubenswrapper[5047]: W1030 20:10:20.238058 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921657da_0e67_403c_8841_7dcaf528268c.slice/crio-d0f312abe4fe678c80ea130d58a79713563ac96ba7714cce0ccd28c0859deae4 WatchSource:0}: Error finding container d0f312abe4fe678c80ea130d58a79713563ac96ba7714cce0ccd28c0859deae4: Status 404 returned error can't find the container with id d0f312abe4fe678c80ea130d58a79713563ac96ba7714cce0ccd28c0859deae4 Oct 30 20:10:20.239518 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:20.239495 5047 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:10:20.794230 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:20.794193 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" event={"ID":"921657da-0e67-403c-8841-7dcaf528268c","Type":"ContainerStarted","Data":"988b7cdd3dda8925b2198afe62ef4dc189ede23100f2e097f9e329263f1d4c7b"} Oct 30 20:10:20.794658 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:20.794247 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" event={"ID":"921657da-0e67-403c-8841-7dcaf528268c","Type":"ContainerStarted","Data":"d0f312abe4fe678c80ea130d58a79713563ac96ba7714cce0ccd28c0859deae4"} Oct 30 20:10:20.794658 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:20.794332 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:10:20.833516 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:20.833455 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" podStartSLOduration=1.481251606 podStartE2EDuration="1.833440461s" totalImagesPullingTime="352.188855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:20.832497471 +0000 UTC m=+3973.906952829" watchObservedRunningTime="2026-10-30 20:10:20.833440461 +0000 UTC m=+3973.907895848" Oct 30 20:10:26.805895 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:10:26.805861 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:10:39.605684 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:10:39.605644 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:11:09.601146 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:11:09.601101 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:11:11.680659 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:11.680625 5047 scope.go:122] "RemoveContainer" containerID="180456f5d4efd080f4e155baec917c381194f530a3099162106b9a6f688b9aaf" Oct 30 20:11:33.357571 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:33.357533 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz"] Oct 30 20:11:33.358083 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:33.357743 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" podUID="921657da-0e67-403c-8841-7dcaf528268c" containerName="maas-api" containerID="cri-o://988b7cdd3dda8925b2198afe62ef4dc189ede23100f2e097f9e329263f1d4c7b" gracePeriod=30 Oct 30 20:11:33.868751 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:33.868725 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:11:33.966594 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:33.966507 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/921657da-0e67-403c-8841-7dcaf528268c-kube-api-access-jkf8d\" (UniqueName: \"kubernetes.io/projected/921657da-0e67-403c-8841-7dcaf528268c-kube-api-access-jkf8d\") pod \"921657da-0e67-403c-8841-7dcaf528268c\" (UID: \"921657da-0e67-403c-8841-7dcaf528268c\") " Oct 30 20:11:33.966769 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:33.966608 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/921657da-0e67-403c-8841-7dcaf528268c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/921657da-0e67-403c-8841-7dcaf528268c-openshift-service-ca\") pod \"921657da-0e67-403c-8841-7dcaf528268c\" (UID: \"921657da-0e67-403c-8841-7dcaf528268c\") " Oct 30 20:11:33.966769 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:33.966650 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/921657da-0e67-403c-8841-7dcaf528268c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/921657da-0e67-403c-8841-7dcaf528268c-maas-api-tls\") pod \"921657da-0e67-403c-8841-7dcaf528268c\" (UID: \"921657da-0e67-403c-8841-7dcaf528268c\") " Oct 30 20:11:33.966769 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:33.966709 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/921657da-0e67-403c-8841-7dcaf528268c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/921657da-0e67-403c-8841-7dcaf528268c-metrics-certs\") pod \"921657da-0e67-403c-8841-7dcaf528268c\" (UID: \"921657da-0e67-403c-8841-7dcaf528268c\") " Oct 30 20:11:33.967027 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:33.966988 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/921657da-0e67-403c-8841-7dcaf528268c-openshift-service-ca" pod "921657da-0e67-403c-8841-7dcaf528268c" (UID: "921657da-0e67-403c-8841-7dcaf528268c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:33.967165 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:33.967142 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/921657da-0e67-403c-8841-7dcaf528268c-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:11:33.968953 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:33.968920 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/921657da-0e67-403c-8841-7dcaf528268c-maas-api-tls" pod "921657da-0e67-403c-8841-7dcaf528268c" (UID: "921657da-0e67-403c-8841-7dcaf528268c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:33.969081 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:33.968981 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/921657da-0e67-403c-8841-7dcaf528268c-kube-api-access-jkf8d" pod "921657da-0e67-403c-8841-7dcaf528268c" (UID: "921657da-0e67-403c-8841-7dcaf528268c"). InnerVolumeSpecName "kube-api-access-jkf8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:33.969081 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:33.968992 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/921657da-0e67-403c-8841-7dcaf528268c-metrics-certs" pod "921657da-0e67-403c-8841-7dcaf528268c" (UID: "921657da-0e67-403c-8841-7dcaf528268c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:34.070474 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:34.068272 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/921657da-0e67-403c-8841-7dcaf528268c-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:11:34.070474 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:34.068315 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/921657da-0e67-403c-8841-7dcaf528268c-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:11:34.070474 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:34.068330 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkf8d\" (UniqueName: \"kubernetes.io/projected/921657da-0e67-403c-8841-7dcaf528268c-kube-api-access-jkf8d\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:11:34.207610 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:34.207574 5047 generic.go:356] "Generic (PLEG): container finished" podID="921657da-0e67-403c-8841-7dcaf528268c" containerID="988b7cdd3dda8925b2198afe62ef4dc189ede23100f2e097f9e329263f1d4c7b" exitCode=0 Oct 30 20:11:34.207792 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:34.207618 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" Oct 30 20:11:34.207792 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:34.207651 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" event={"ID":"921657da-0e67-403c-8841-7dcaf528268c","Type":"ContainerDied","Data":"988b7cdd3dda8925b2198afe62ef4dc189ede23100f2e097f9e329263f1d4c7b"} Oct 30 20:11:34.207792 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:34.207703 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz" event={"ID":"921657da-0e67-403c-8841-7dcaf528268c","Type":"ContainerDied","Data":"d0f312abe4fe678c80ea130d58a79713563ac96ba7714cce0ccd28c0859deae4"} Oct 30 20:11:34.207792 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:34.207725 5047 scope.go:122] "RemoveContainer" containerID="988b7cdd3dda8925b2198afe62ef4dc189ede23100f2e097f9e329263f1d4c7b" Oct 30 20:11:34.217901 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:34.217882 5047 scope.go:122] "RemoveContainer" containerID="988b7cdd3dda8925b2198afe62ef4dc189ede23100f2e097f9e329263f1d4c7b" Oct 30 20:11:34.218216 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:11:34.218197 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"988b7cdd3dda8925b2198afe62ef4dc189ede23100f2e097f9e329263f1d4c7b\": container with ID starting with 988b7cdd3dda8925b2198afe62ef4dc189ede23100f2e097f9e329263f1d4c7b not found: ID does not exist" containerID="988b7cdd3dda8925b2198afe62ef4dc189ede23100f2e097f9e329263f1d4c7b" Oct 30 20:11:34.218281 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:34.218227 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988b7cdd3dda8925b2198afe62ef4dc189ede23100f2e097f9e329263f1d4c7b"} err="failed to get container status \"988b7cdd3dda8925b2198afe62ef4dc189ede23100f2e097f9e329263f1d4c7b\": rpc error: code = NotFound desc = could not find container \"988b7cdd3dda8925b2198afe62ef4dc189ede23100f2e097f9e329263f1d4c7b\": container with ID starting with 988b7cdd3dda8925b2198afe62ef4dc189ede23100f2e097f9e329263f1d4c7b not found: ID does not exist" Oct 30 20:11:34.255646 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:34.255605 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz"] Oct 30 20:11:34.257654 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:34.257632 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-013647-68b55684db-4tglz"] Oct 30 20:11:35.055761 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:11:35.055720 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="921657da-0e67-403c-8841-7dcaf528268c" path="/var/lib/kubelet/pods/921657da-0e67-403c-8841-7dcaf528268c/volumes" Oct 30 20:11:39.601586 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:11:39.601548 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:12:09.602839 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:12:09.602800 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:12:39.601155 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:12:39.601063 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:13:09.600970 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:13:09.600925 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:13:39.601638 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:13:39.601599 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:14:08.068370 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:08.068328 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 20:14:08.078425 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:08.078399 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 20:14:09.603135 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:14:09.603096 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:14:23.780365 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.780328 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4"] Oct 30 20:14:23.780860 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.780798 5047 cpu_manager.go:414] "containerMap: removing container" podUID="921657da-0e67-403c-8841-7dcaf528268c" containerName="maas-api" Oct 30 20:14:23.780860 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.780810 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="921657da-0e67-403c-8841-7dcaf528268c" containerName="maas-api" Oct 30 20:14:23.780942 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.780887 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="921657da-0e67-403c-8841-7dcaf528268c" containerName="maas-api" Oct 30 20:14:23.784635 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.784603 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:14:23.799313 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.799284 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-0bdca7-serving-cert\"" Oct 30 20:14:23.813519 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.813483 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4"] Oct 30 20:14:23.886453 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.886400 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4\" (UID: \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:14:23.886681 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.886480 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4\" (UID: \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:14:23.886681 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.886556 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4\" (UID: \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:14:23.886814 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.886708 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpsnh\" (UniqueName: \"kubernetes.io/projected/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-kube-api-access-gpsnh\") pod \"maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4\" (UID: \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:14:23.987754 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.987719 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4\" (UID: \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:14:23.987967 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.987770 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4\" (UID: \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:14:23.987967 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.987796 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4\" (UID: \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:14:23.987967 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.987864 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpsnh\" (UniqueName: \"kubernetes.io/projected/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-kube-api-access-gpsnh\") pod \"maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4\" (UID: \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:14:23.988619 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.988588 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4\" (UID: \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:14:23.990381 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.990360 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-metrics-certs\") pod \"maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4\" (UID: \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:14:23.990489 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:23.990467 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4\" (UID: \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:14:24.011798 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:24.011760 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpsnh\" (UniqueName: \"kubernetes.io/projected/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-kube-api-access-gpsnh\") pod \"maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4\" (UID: \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:14:24.100189 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:24.100136 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:14:24.413695 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:24.413660 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4"] Oct 30 20:14:24.415021 ip-10-0-140-235 kubenswrapper[5047]: W1030 20:14:24.414982 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2a0bc5_5a9c_4ba5_8cd4_aa6e9f4973f0.slice/crio-d9f508db7367f5b17b482b5accccc25dce381e5ad4452e45a55e01b49efe312e WatchSource:0}: Error finding container d9f508db7367f5b17b482b5accccc25dce381e5ad4452e45a55e01b49efe312e: Status 404 returned error can't find the container with id d9f508db7367f5b17b482b5accccc25dce381e5ad4452e45a55e01b49efe312e Oct 30 20:14:25.365695 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:25.365660 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" event={"ID":"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0","Type":"ContainerStarted","Data":"6b09e3a69213943444d6fd8ede8610bdfddbc4580ca389d55de114c050c44499"} Oct 30 20:14:25.366188 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:25.365713 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" event={"ID":"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0","Type":"ContainerStarted","Data":"d9f508db7367f5b17b482b5accccc25dce381e5ad4452e45a55e01b49efe312e"} Oct 30 20:14:25.366188 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:25.365795 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:14:25.406875 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:25.406820 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" podStartSLOduration=2.097743847 podStartE2EDuration="2.406805702s" totalImagesPullingTime="309.061855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:25.406031474 +0000 UTC m=+4218.480486830" watchObservedRunningTime="2026-10-30 20:14:25.406805702 +0000 UTC m=+4218.481261061" Oct 30 20:14:31.376440 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:14:31.376404 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:14:39.601887 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:14:39.601845 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:15:09.600787 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:15:09.600744 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:15:39.601089 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:15:39.600986 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:15:53.368994 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:53.368954 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4"] Oct 30 20:15:53.369528 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:53.369208 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" podUID="6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0" containerName="maas-api" containerID="cri-o://6b09e3a69213943444d6fd8ede8610bdfddbc4580ca389d55de114c050c44499" gracePeriod=30 Oct 30 20:15:53.848347 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:53.848313 5047 generic.go:356] "Generic (PLEG): container finished" podID="6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0" containerID="6b09e3a69213943444d6fd8ede8610bdfddbc4580ca389d55de114c050c44499" exitCode=0 Oct 30 20:15:53.848500 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:53.848393 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" event={"ID":"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0","Type":"ContainerDied","Data":"6b09e3a69213943444d6fd8ede8610bdfddbc4580ca389d55de114c050c44499"} Oct 30 20:15:53.848500 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:53.848460 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" event={"ID":"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0","Type":"ContainerDied","Data":"d9f508db7367f5b17b482b5accccc25dce381e5ad4452e45a55e01b49efe312e"} Oct 30 20:15:53.848500 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:53.848483 5047 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9f508db7367f5b17b482b5accccc25dce381e5ad4452e45a55e01b49efe312e" Oct 30 20:15:53.854753 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:53.854732 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:15:53.952574 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:53.952477 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-maas-api-tls\") pod \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\" (UID: \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\") " Oct 30 20:15:53.952574 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:53.952561 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-metrics-certs\") pod \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\" (UID: \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\") " Oct 30 20:15:53.952826 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:53.952625 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-openshift-service-ca\") pod \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\" (UID: \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\") " Oct 30 20:15:53.952826 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:53.952670 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-kube-api-access-gpsnh\" (UniqueName: \"kubernetes.io/projected/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-kube-api-access-gpsnh\") pod \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\" (UID: \"6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0\") " Oct 30 20:15:53.953053 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:53.953023 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-openshift-service-ca" pod "6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0" (UID: "6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:53.954938 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:53.954907 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-kube-api-access-gpsnh" pod "6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0" (UID: "6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0"). InnerVolumeSpecName "kube-api-access-gpsnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:53.955201 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:53.955178 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-maas-api-tls" pod "6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0" (UID: "6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:53.955282 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:53.955194 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-metrics-certs" pod "6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0" (UID: "6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:54.053451 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:54.053421 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:15:54.053451 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:54.053447 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpsnh\" (UniqueName: \"kubernetes.io/projected/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-kube-api-access-gpsnh\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:15:54.053451 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:54.053457 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:15:54.053691 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:54.053466 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:15:54.863846 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:54.863798 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4" Oct 30 20:15:54.913211 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:54.913174 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4"] Oct 30 20:15:54.924661 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:54.924630 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-0bdca7-945fd5c8-bn2z4"] Oct 30 20:15:55.047948 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:15:55.047912 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0" path="/var/lib/kubelet/pods/6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0/volumes" Oct 30 20:16:09.600812 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:16:09.600769 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:16:39.600460 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:16:39.600419 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:17:09.600860 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:17:09.600818 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:17:38.677031 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:17:38.676230 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xljwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:17:38.677031 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:17:38.676308 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:17:38.730177 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:17:38.730131 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:17:38.730530 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:17:38.730498 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:17:39.601518 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:17:39.601481 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:18:00.643032 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:18:00.639921 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-frwdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:18:00.643032 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:18:00.639999 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" podUID="5fcb783a-2210-41a3-8379-b2c417ac94d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:18:08.680154 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:18:08.680112 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xljwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:18:08.680773 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:18:08.680739 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:18:08.735165 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:18:08.735105 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:18:08.735500 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:18:08.735469 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:18:09.600140 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:18:09.600103 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:18:23.685088 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:18:23.682133 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-xljwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:18:23.685088 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:18:23.682231 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:18:23.722776 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:18:23.722747 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-bb2r4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:18:23.723031 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:18:23.722991 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:18:39.599409 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:18:39.599372 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:19:08.108155 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.108123 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 20:19:08.116744 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.116713 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 20:19:08.562065 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.561981 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-p6jwg"] Oct 30 20:19:08.562699 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.562675 5047 cpu_manager.go:414] "containerMap: removing container" podUID="6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0" containerName="maas-api" Oct 30 20:19:08.562699 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.562699 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0" containerName="maas-api" Oct 30 20:19:08.562910 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.562829 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a2a0bc5-5a9c-4ba5-8cd4-aa6e9f4973f0" containerName="maas-api" Oct 30 20:19:08.566258 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.566238 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" Oct 30 20:19:08.573506 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.573332 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/269993bc-c3b7-4e8d-9dab-68c120b0a881-openshift-service-ca\") pod \"maas-api-595cc648c-p6jwg\" (UID: \"269993bc-c3b7-4e8d-9dab-68c120b0a881\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" Oct 30 20:19:08.573506 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.573408 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/269993bc-c3b7-4e8d-9dab-68c120b0a881-metrics-certs\") pod \"maas-api-595cc648c-p6jwg\" (UID: \"269993bc-c3b7-4e8d-9dab-68c120b0a881\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" Oct 30 20:19:08.573629 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.573517 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kl2q\" (UniqueName: \"kubernetes.io/projected/269993bc-c3b7-4e8d-9dab-68c120b0a881-kube-api-access-8kl2q\") pod \"maas-api-595cc648c-p6jwg\" (UID: \"269993bc-c3b7-4e8d-9dab-68c120b0a881\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" Oct 30 20:19:08.573629 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.573583 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/269993bc-c3b7-4e8d-9dab-68c120b0a881-maas-api-tls\") pod \"maas-api-595cc648c-p6jwg\" (UID: \"269993bc-c3b7-4e8d-9dab-68c120b0a881\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" Oct 30 20:19:08.595525 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.595486 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-p6jwg"] Oct 30 20:19:08.674670 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.674630 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/269993bc-c3b7-4e8d-9dab-68c120b0a881-maas-api-tls\") pod \"maas-api-595cc648c-p6jwg\" (UID: \"269993bc-c3b7-4e8d-9dab-68c120b0a881\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" Oct 30 20:19:08.674845 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.674735 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/269993bc-c3b7-4e8d-9dab-68c120b0a881-openshift-service-ca\") pod \"maas-api-595cc648c-p6jwg\" (UID: \"269993bc-c3b7-4e8d-9dab-68c120b0a881\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" Oct 30 20:19:08.674845 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.674781 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/269993bc-c3b7-4e8d-9dab-68c120b0a881-metrics-certs\") pod \"maas-api-595cc648c-p6jwg\" (UID: \"269993bc-c3b7-4e8d-9dab-68c120b0a881\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" Oct 30 20:19:08.674845 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.674827 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kl2q\" (UniqueName: \"kubernetes.io/projected/269993bc-c3b7-4e8d-9dab-68c120b0a881-kube-api-access-8kl2q\") pod \"maas-api-595cc648c-p6jwg\" (UID: \"269993bc-c3b7-4e8d-9dab-68c120b0a881\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" Oct 30 20:19:08.675618 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.675593 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/269993bc-c3b7-4e8d-9dab-68c120b0a881-openshift-service-ca\") pod \"maas-api-595cc648c-p6jwg\" (UID: \"269993bc-c3b7-4e8d-9dab-68c120b0a881\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" Oct 30 20:19:08.677222 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.677197 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/269993bc-c3b7-4e8d-9dab-68c120b0a881-metrics-certs\") pod \"maas-api-595cc648c-p6jwg\" (UID: \"269993bc-c3b7-4e8d-9dab-68c120b0a881\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" Oct 30 20:19:08.677353 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.677331 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/269993bc-c3b7-4e8d-9dab-68c120b0a881-maas-api-tls\") pod \"maas-api-595cc648c-p6jwg\" (UID: \"269993bc-c3b7-4e8d-9dab-68c120b0a881\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" Oct 30 20:19:08.693724 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.693688 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kl2q\" (UniqueName: \"kubernetes.io/projected/269993bc-c3b7-4e8d-9dab-68c120b0a881-kube-api-access-8kl2q\") pod \"maas-api-595cc648c-p6jwg\" (UID: \"269993bc-c3b7-4e8d-9dab-68c120b0a881\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" Oct 30 20:19:08.879954 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:08.879910 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" Oct 30 20:19:09.178997 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:09.178969 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-p6jwg"] Oct 30 20:19:09.180220 ip-10-0-140-235 kubenswrapper[5047]: W1030 20:19:09.180190 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269993bc_c3b7_4e8d_9dab_68c120b0a881.slice/crio-d1dd338e741bb3a9446bc971d4b1bd80a54b3a8b9adbc3c01c13ae5aef03c53c WatchSource:0}: Error finding container d1dd338e741bb3a9446bc971d4b1bd80a54b3a8b9adbc3c01c13ae5aef03c53c: Status 404 returned error can't find the container with id d1dd338e741bb3a9446bc971d4b1bd80a54b3a8b9adbc3c01c13ae5aef03c53c Oct 30 20:19:09.181555 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:09.181527 5047 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:19:09.602016 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:19:09.601967 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:19:10.120977 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:10.120931 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" event={"ID":"269993bc-c3b7-4e8d-9dab-68c120b0a881","Type":"ContainerStarted","Data":"cb03168d9c1ba51dffe31ac4cce0dd931c3573e859e3158ee96c8c3662eae897"} Oct 30 20:19:10.121188 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:10.120998 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" event={"ID":"269993bc-c3b7-4e8d-9dab-68c120b0a881","Type":"ContainerStarted","Data":"d1dd338e741bb3a9446bc971d4b1bd80a54b3a8b9adbc3c01c13ae5aef03c53c"} Oct 30 20:19:10.121188 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:10.121128 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" Oct 30 20:19:10.161593 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:10.161541 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" podStartSLOduration=1.772716987 podStartE2EDuration="2.161528461s" totalImagesPullingTime="388.811474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:10.160778613 +0000 UTC m=+4503.235233972" watchObservedRunningTime="2026-10-30 20:19:10.161528461 +0000 UTC m=+4503.235983820" Oct 30 20:19:16.132373 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:16.132341 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-p6jwg" Oct 30 20:19:25.358761 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:25.358722 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c"] Oct 30 20:19:25.359637 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:25.359580 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" podUID="b0d053db-3c2e-4701-b045-68a75abeaa89" containerName="maas-api" containerID="cri-o://100b47dfb8ebd1de2e3239f1c18206449c7e7b4af5676cfbd3693a47bd998471" gracePeriod=30 Oct 30 20:19:25.632946 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:25.632855 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" podUID="b0d053db-3c2e-4701-b045-68a75abeaa89" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.44:8443/health\": dial tcp 10.132.0.44:8443: connect: connection refused" Oct 30 20:19:26.626186 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:26.626156 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 20:19:26.734441 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:26.734354 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0d053db-3c2e-4701-b045-68a75abeaa89-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0d053db-3c2e-4701-b045-68a75abeaa89-metrics-certs\") pod \"b0d053db-3c2e-4701-b045-68a75abeaa89\" (UID: \"b0d053db-3c2e-4701-b045-68a75abeaa89\") " Oct 30 20:19:26.734441 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:26.734397 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d053db-3c2e-4701-b045-68a75abeaa89-kube-api-access-2c62k\" (UniqueName: \"kubernetes.io/projected/b0d053db-3c2e-4701-b045-68a75abeaa89-kube-api-access-2c62k\") pod \"b0d053db-3c2e-4701-b045-68a75abeaa89\" (UID: \"b0d053db-3c2e-4701-b045-68a75abeaa89\") " Oct 30 20:19:26.734715 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:26.734460 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0d053db-3c2e-4701-b045-68a75abeaa89-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0d053db-3c2e-4701-b045-68a75abeaa89-maas-api-tls\") pod \"b0d053db-3c2e-4701-b045-68a75abeaa89\" (UID: \"b0d053db-3c2e-4701-b045-68a75abeaa89\") " Oct 30 20:19:26.734715 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:26.734480 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0d053db-3c2e-4701-b045-68a75abeaa89-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0d053db-3c2e-4701-b045-68a75abeaa89-openshift-service-ca\") pod \"b0d053db-3c2e-4701-b045-68a75abeaa89\" (UID: \"b0d053db-3c2e-4701-b045-68a75abeaa89\") " Oct 30 20:19:26.735032 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:26.734970 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d053db-3c2e-4701-b045-68a75abeaa89-openshift-service-ca" pod "b0d053db-3c2e-4701-b045-68a75abeaa89" (UID: "b0d053db-3c2e-4701-b045-68a75abeaa89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:26.736683 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:26.736653 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d053db-3c2e-4701-b045-68a75abeaa89-maas-api-tls" pod "b0d053db-3c2e-4701-b045-68a75abeaa89" (UID: "b0d053db-3c2e-4701-b045-68a75abeaa89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:19:26.736683 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:26.736669 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d053db-3c2e-4701-b045-68a75abeaa89-metrics-certs" pod "b0d053db-3c2e-4701-b045-68a75abeaa89" (UID: "b0d053db-3c2e-4701-b045-68a75abeaa89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:19:26.736827 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:26.736663 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d053db-3c2e-4701-b045-68a75abeaa89-kube-api-access-2c62k" pod "b0d053db-3c2e-4701-b045-68a75abeaa89" (UID: "b0d053db-3c2e-4701-b045-68a75abeaa89"). InnerVolumeSpecName "kube-api-access-2c62k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:26.836521 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:26.836481 5047 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0d053db-3c2e-4701-b045-68a75abeaa89-maas-api-tls\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:19:26.836521 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:26.836518 5047 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0d053db-3c2e-4701-b045-68a75abeaa89-openshift-service-ca\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:19:26.836521 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:26.836527 5047 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0d053db-3c2e-4701-b045-68a75abeaa89-metrics-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:19:26.836521 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:26.836536 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c62k\" (UniqueName: \"kubernetes.io/projected/b0d053db-3c2e-4701-b045-68a75abeaa89-kube-api-access-2c62k\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:19:27.258625 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:27.258589 5047 generic.go:356] "Generic (PLEG): container finished" podID="b0d053db-3c2e-4701-b045-68a75abeaa89" containerID="100b47dfb8ebd1de2e3239f1c18206449c7e7b4af5676cfbd3693a47bd998471" exitCode=0 Oct 30 20:19:27.258808 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:27.258638 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" Oct 30 20:19:27.258808 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:27.258654 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" event={"ID":"b0d053db-3c2e-4701-b045-68a75abeaa89","Type":"ContainerDied","Data":"100b47dfb8ebd1de2e3239f1c18206449c7e7b4af5676cfbd3693a47bd998471"} Oct 30 20:19:27.258808 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:27.258696 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c" event={"ID":"b0d053db-3c2e-4701-b045-68a75abeaa89","Type":"ContainerDied","Data":"ef4689abcc9451f296ae9c37641fe16095c1338075b1069417e0c19dc5ec2228"} Oct 30 20:19:27.258808 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:27.258718 5047 scope.go:122] "RemoveContainer" containerID="100b47dfb8ebd1de2e3239f1c18206449c7e7b4af5676cfbd3693a47bd998471" Oct 30 20:19:27.268038 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:27.267999 5047 scope.go:122] "RemoveContainer" containerID="100b47dfb8ebd1de2e3239f1c18206449c7e7b4af5676cfbd3693a47bd998471" Oct 30 20:19:27.268362 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:19:27.268338 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"100b47dfb8ebd1de2e3239f1c18206449c7e7b4af5676cfbd3693a47bd998471\": container with ID starting with 100b47dfb8ebd1de2e3239f1c18206449c7e7b4af5676cfbd3693a47bd998471 not found: ID does not exist" containerID="100b47dfb8ebd1de2e3239f1c18206449c7e7b4af5676cfbd3693a47bd998471" Oct 30 20:19:27.268413 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:27.268374 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"100b47dfb8ebd1de2e3239f1c18206449c7e7b4af5676cfbd3693a47bd998471"} err="failed to get container status \"100b47dfb8ebd1de2e3239f1c18206449c7e7b4af5676cfbd3693a47bd998471\": rpc error: code = NotFound desc = could not find container \"100b47dfb8ebd1de2e3239f1c18206449c7e7b4af5676cfbd3693a47bd998471\": container with ID starting with 100b47dfb8ebd1de2e3239f1c18206449c7e7b4af5676cfbd3693a47bd998471 not found: ID does not exist" Oct 30 20:19:27.306540 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:27.306452 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c"] Oct 30 20:19:27.317693 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:27.317660 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-jlg7c"] Oct 30 20:19:29.056144 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:19:29.056103 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d053db-3c2e-4701-b045-68a75abeaa89" path="/var/lib/kubelet/pods/b0d053db-3c2e-4701-b045-68a75abeaa89/volumes" Oct 30 20:19:39.604382 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:19:39.604342 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:20:09.602042 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:20:09.601987 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:20:35.868331 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:20:35.868293 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-84dd667bd6-hz7tl_3977e82d-a7b6-4f25-aa74-8d1f6488ff41/manager/0.log" Oct 30 20:20:36.905739 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:20:36.905703 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65cb454cc4-5lnl7_70ddcddf-b107-4a4a-ac56-4b451ead2612/manager/0.log" Oct 30 20:20:37.633918 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:20:37.633880 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-564b7_0f369006-b3bf-412c-9c5b-90e2f945bc76/manager/0.log" Oct 30 20:20:38.086671 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:20:38.086636 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-ch865_dfe5c738-462f-461b-b1e9-1f449896ed4e/copy-manifests/0.log" Oct 30 20:20:38.118716 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:20:38.118681 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-ch865_dfe5c738-462f-461b-b1e9-1f449896ed4e/manager/0.log" Oct 30 20:20:38.618809 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:20:38.618781 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-vz8vl_a703457c-cd6f-4b39-b8db-e7c3c419e72d/proxy/0.log" Oct 30 20:20:39.600846 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:20:39.600811 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:20:39.829523 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:20:39.829485 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-p6jwg_269993bc-c3b7-4e8d-9dab-68c120b0a881/maas-api/0.log" Oct 30 20:20:41.178957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:20:41.178922 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-hmblk_812cd787-806e-4ca5-8d92-e53ad0581e8c/postgres/0.log" Oct 30 20:20:42.979765 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:20:42.979731 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-kdv7t_d7f89e1b-5ac7-46ce-9a5b-77a1ec2861b5/manager/0.log" Oct 30 20:20:44.018023 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:20:44.017971 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-nm5kw_369870ff-3389-4bb6-8b76-05a35b0cc02b/manager/0.log" Oct 30 20:20:45.007623 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:20:45.007587 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kzmjv_7d62bc31-309d-4724-8bda-83f24f229a2e/discovery/0.log" Oct 30 20:20:45.401988 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:20:45.401950 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c7d47d6c9-jvqgk_eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d/kube-auth-proxy/0.log" Oct 30 20:20:45.806429 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:20:45.806343 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-bb2r4_2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb/istio-proxy/0.log" Oct 30 20:20:46.411562 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:20:46.411525 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-frwdk_5fcb783a-2210-41a3-8379-b2c417ac94d8/istio-proxy/0.log" Oct 30 20:20:46.840077 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:20:46.840046 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-mpgxb_dbda9b5f-24c3-4cbc-ba7f-d134eda7adce/istio-proxy/0.log" Oct 30 20:20:57.469402 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:20:57.469354 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-xljwk_5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4/istio-proxy/0.log" Oct 30 20:21:07.700265 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:07.700226 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-mptvz_645e5088-e265-4836-a447-1cb2b21cc050/payload-pre-processing/0.log" Oct 30 20:21:07.912772 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:07.912735 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rspbb_8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a/payload-processing/0.log" Oct 30 20:21:08.153673 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:08.153641 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77d478f789-d787c_20403b1e-e479-45b9-bb85-52cf10725d86/router/0.log" Oct 30 20:21:09.194166 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:09.194131 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-5bffn_c276eb19-3a26-4138-9545-d465361783f6/main/0.log" Oct 30 20:21:09.600718 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:21:09.600682 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:21:09.812895 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:09.812859 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg_d6cf3827-a5fe-48e4-900a-1f1633ff69ca/main/0.log" Oct 30 20:21:10.019455 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:10.019368 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g_1b105e8c-2f71-484d-b6fa-454fdc1dc3e2/main/0.log" Oct 30 20:21:11.782111 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:11.782077 5047 scope.go:122] "RemoveContainer" containerID="6b09e3a69213943444d6fd8ede8610bdfddbc4580ca389d55de114c050c44499" Oct 30 20:21:30.419328 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:30.419235 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-84dd667bd6-hz7tl_3977e82d-a7b6-4f25-aa74-8d1f6488ff41/manager/0.log" Oct 30 20:21:31.370001 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:31.369958 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65cb454cc4-5lnl7_70ddcddf-b107-4a4a-ac56-4b451ead2612/manager/0.log" Oct 30 20:21:39.599973 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:21:39.599931 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:21:42.095155 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:42.095117 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-564b7_0f369006-b3bf-412c-9c5b-90e2f945bc76/manager/0.log" Oct 30 20:21:42.597481 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:42.597449 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-ch865_dfe5c738-462f-461b-b1e9-1f449896ed4e/manager/0.log" Oct 30 20:21:42.629019 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:42.628981 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-ch865_dfe5c738-462f-461b-b1e9-1f449896ed4e/copy-manifests/0.log" Oct 30 20:21:43.119666 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:43.119630 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-vz8vl_a703457c-cd6f-4b39-b8db-e7c3c419e72d/proxy/0.log" Oct 30 20:21:53.881189 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:53.881145 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-p6jwg_269993bc-c3b7-4e8d-9dab-68c120b0a881/maas-api/0.log" Oct 30 20:21:55.287370 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:55.287338 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-hmblk_812cd787-806e-4ca5-8d92-e53ad0581e8c/postgres/0.log" Oct 30 20:21:56.728450 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:56.728416 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-kdv7t_d7f89e1b-5ac7-46ce-9a5b-77a1ec2861b5/manager/0.log" Oct 30 20:21:57.788240 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:57.788203 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-nm5kw_369870ff-3389-4bb6-8b76-05a35b0cc02b/manager/0.log" Oct 30 20:21:58.606169 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:58.606135 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kzmjv_7d62bc31-309d-4724-8bda-83f24f229a2e/discovery/0.log" Oct 30 20:21:59.009289 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:59.009209 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c7d47d6c9-jvqgk_eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d/kube-auth-proxy/0.log" Oct 30 20:21:59.403335 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:21:59.403304 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-bb2r4_2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb/istio-proxy/0.log" Oct 30 20:22:00.014493 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:22:00.014461 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-frwdk_5fcb783a-2210-41a3-8379-b2c417ac94d8/istio-proxy/0.log" Oct 30 20:22:00.442542 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:22:00.442503 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-mpgxb_dbda9b5f-24c3-4cbc-ba7f-d134eda7adce/istio-proxy/0.log" Oct 30 20:22:01.055566 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:22:01.055532 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-xljwk_5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4/istio-proxy/0.log" Oct 30 20:22:01.265209 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:22:01.265176 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-mptvz_645e5088-e265-4836-a447-1cb2b21cc050/payload-pre-processing/0.log" Oct 30 20:22:01.455484 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:22:01.455451 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rspbb_8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a/payload-processing/0.log" Oct 30 20:22:01.658947 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:22:01.658903 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77d478f789-d787c_20403b1e-e479-45b9-bb85-52cf10725d86/router/0.log" Oct 30 20:22:02.415726 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:22:02.415696 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-5bffn_c276eb19-3a26-4138-9545-d465361783f6/main/0.log" Oct 30 20:22:09.601518 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:22:09.601477 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:22:13.021917 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:22:13.021882 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dlsrg_d6cf3827-a5fe-48e4-900a-1f1633ff69ca/main/0.log" Oct 30 20:22:13.240157 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:22:13.240125 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-4kt2g_1b105e8c-2f71-484d-b6fa-454fdc1dc3e2/main/0.log" Oct 30 20:22:39.600974 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:22:39.600931 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:23:09.601034 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:23:09.600965 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:23:14.160173 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:23:14.160137 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65cb454cc4-5lnl7_70ddcddf-b107-4a4a-ac56-4b451ead2612/manager/0.log" Oct 30 20:23:14.495557 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:23:14.495451 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65cb454cc4-5lnl7_70ddcddf-b107-4a4a-ac56-4b451ead2612/manager/0.log" Oct 30 20:23:15.188418 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:23:15.188389 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rspbb_8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a/payload-processing/0.log" Oct 30 20:23:25.373135 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:23:25.373096 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rspbb_8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a/payload-processing/0.log" Oct 30 20:23:39.600612 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:23:39.600543 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:24:08.147807 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:24:08.147776 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 20:24:08.154993 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:24:08.154968 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 20:24:09.600312 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:24:09.600276 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:24:39.602220 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:24:39.602132 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:24:54.189390 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:24:54.189356 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65cb454cc4-5lnl7_70ddcddf-b107-4a4a-ac56-4b451ead2612/manager/0.log" Oct 30 20:24:54.538268 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:24:54.538188 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65cb454cc4-5lnl7_70ddcddf-b107-4a4a-ac56-4b451ead2612/manager/0.log" Oct 30 20:24:55.241915 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:24:55.241883 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rspbb_8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a/payload-processing/0.log" Oct 30 20:24:55.488088 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:24:55.488058 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rspbb_8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a/payload-processing/0.log" Oct 30 20:25:09.599708 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:25:09.599667 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:25:10.153177 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:10.153149 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hx495_ad4cd78b-b491-4927-80ae-dae026d10e7a/global-pull-secret-syncer/0.log" Oct 30 20:25:10.527631 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:10.527547 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-fs89c_a8d10e22-9215-4fc2-8474-5bc68cb57f18/connection-checker/0.log" Oct 30 20:25:10.646028 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:10.645977 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-pqg4r_57ee29d6-ad4d-4ed8-ae1a-877732a3737b/connection-checker/0.log" Oct 30 20:25:10.773350 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:10.773323 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-sfbbt_8ae35879-1912-43b0-a7a5-9ac064ce0dea/connection-checker/0.log" Oct 30 20:25:11.082126 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:11.082092 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d26nk_633f6b63-a067-470b-bec9-e913e7f28589/konnectivity-agent/0.log" Oct 30 20:25:11.546053 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:11.545962 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-235.ec2.internal_b021a22e7072822cf09556c5f1f04826/haproxy/0.log" Oct 30 20:25:28.866275 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:28.866238 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk"] Oct 30 20:25:28.866800 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:28.866503 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" podUID="5fcb783a-2210-41a3-8379-b2c417ac94d8" containerName="istio-proxy" containerID="cri-o://8f20a68f61c38434afa64fb51ff4b7758d2f38f6f132f02915d971141f61a0dd" gracePeriod=30 Oct 30 20:25:28.877834 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:28.877798 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk"] Oct 30 20:25:28.878073 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:28.878043 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerName="istio-proxy" containerID="cri-o://cc0f72e74a64faad186056bbd980546c6e6021bb54d755d1d0fe52b13f4a52a5" gracePeriod=30 Oct 30 20:25:28.880467 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:28.880442 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb"] Oct 30 20:25:28.881034 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:28.880972 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" podUID="dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" containerName="istio-proxy" containerID="cri-o://0f2a8b23795e8ef06cf2df563cc29498c02ad9a295d5c3c781afcd086e4bc8c8" gracePeriod=30 Oct 30 20:25:28.889401 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:28.889375 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4"] Oct 30 20:25:28.889553 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:28.889533 5047 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerName="istio-proxy" containerID="cri-o://7cd34f89f7de6a092610ea8f6db53a6b8875a25a94e797a0f519993e4bb3a63a" gracePeriod=30 Oct 30 20:25:29.591645 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:29.591606 5047 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-frwdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:25:29.591822 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:29.591676 5047 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" podUID="5fcb783a-2210-41a3-8379-b2c417ac94d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:25:34.163726 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.163527 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:25:34.210962 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.210923 5047 generic.go:356] "Generic (PLEG): container finished" podID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerID="7cd34f89f7de6a092610ea8f6db53a6b8875a25a94e797a0f519993e4bb3a63a" exitCode=0 Oct 30 20:25:34.211159 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.211136 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" event={"ID":"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb","Type":"ContainerDied","Data":"7cd34f89f7de6a092610ea8f6db53a6b8875a25a94e797a0f519993e4bb3a63a"} Oct 30 20:25:34.213544 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.213520 5047 generic.go:356] "Generic (PLEG): container finished" podID="dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" containerID="0f2a8b23795e8ef06cf2df563cc29498c02ad9a295d5c3c781afcd086e4bc8c8" exitCode=0 Oct 30 20:25:34.213669 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.213579 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" event={"ID":"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce","Type":"ContainerDied","Data":"0f2a8b23795e8ef06cf2df563cc29498c02ad9a295d5c3c781afcd086e4bc8c8"} Oct 30 20:25:34.215254 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.215229 5047 generic.go:356] "Generic (PLEG): container finished" podID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerID="cc0f72e74a64faad186056bbd980546c6e6021bb54d755d1d0fe52b13f4a52a5" exitCode=0 Oct 30 20:25:34.215367 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.215290 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" Oct 30 20:25:34.215367 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.215315 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" event={"ID":"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4","Type":"ContainerDied","Data":"cc0f72e74a64faad186056bbd980546c6e6021bb54d755d1d0fe52b13f4a52a5"} Oct 30 20:25:34.215462 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.215383 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk" event={"ID":"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4","Type":"ContainerDied","Data":"fac33d4f408f89a3e2207a13e026d29d253a071866bb4c499ce0b553d887f826"} Oct 30 20:25:34.215462 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.215422 5047 scope.go:122] "RemoveContainer" containerID="cc0f72e74a64faad186056bbd980546c6e6021bb54d755d1d0fe52b13f4a52a5" Oct 30 20:25:34.216973 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.216948 5047 generic.go:356] "Generic (PLEG): container finished" podID="5fcb783a-2210-41a3-8379-b2c417ac94d8" containerID="8f20a68f61c38434afa64fb51ff4b7758d2f38f6f132f02915d971141f61a0dd" exitCode=0 Oct 30 20:25:34.217147 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.217017 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" event={"ID":"5fcb783a-2210-41a3-8379-b2c417ac94d8","Type":"ContainerDied","Data":"8f20a68f61c38434afa64fb51ff4b7758d2f38f6f132f02915d971141f61a0dd"} Oct 30 20:25:34.224821 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.224771 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 20:25:34.228723 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.228170 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:25:34.229773 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.229744 5047 scope.go:122] "RemoveContainer" containerID="cc0f72e74a64faad186056bbd980546c6e6021bb54d755d1d0fe52b13f4a52a5" Oct 30 20:25:34.230454 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:25:34.230422 5047 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc0f72e74a64faad186056bbd980546c6e6021bb54d755d1d0fe52b13f4a52a5\": container with ID starting with cc0f72e74a64faad186056bbd980546c6e6021bb54d755d1d0fe52b13f4a52a5 not found: ID does not exist" containerID="cc0f72e74a64faad186056bbd980546c6e6021bb54d755d1d0fe52b13f4a52a5" Oct 30 20:25:34.230571 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.230464 5047 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0f72e74a64faad186056bbd980546c6e6021bb54d755d1d0fe52b13f4a52a5"} err="failed to get container status \"cc0f72e74a64faad186056bbd980546c6e6021bb54d755d1d0fe52b13f4a52a5\": rpc error: code = NotFound desc = could not find container \"cc0f72e74a64faad186056bbd980546c6e6021bb54d755d1d0fe52b13f4a52a5\": container with ID starting with cc0f72e74a64faad186056bbd980546c6e6021bb54d755d1d0fe52b13f4a52a5 not found: ID does not exist" Oct 30 20:25:34.238631 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.238603 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:25:34.334213 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.334181 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-workload-certs\") pod \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " Oct 30 20:25:34.334415 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.334230 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-workload-socket\") pod \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " Oct 30 20:25:34.334415 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.334273 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-envoy\") pod \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " Oct 30 20:25:34.334415 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.334303 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-data\") pod \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " Oct 30 20:25:34.334415 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.334345 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-envoy\") pod \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " Oct 30 20:25:34.334415 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.334378 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-envoy\") pod \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " Oct 30 20:25:34.334415 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.334413 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-data\") pod \"5fcb783a-2210-41a3-8379-b2c417ac94d8\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " Oct 30 20:25:34.334742 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.334443 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-kube-api-access-6mht8\" (UniqueName: \"kubernetes.io/projected/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-kube-api-access-6mht8\") pod \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " Oct 30 20:25:34.334742 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.334470 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-credential-socket\") pod \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " Oct 30 20:25:34.334742 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.334516 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-podinfo\") pod \"5fcb783a-2210-41a3-8379-b2c417ac94d8\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " Oct 30 20:25:34.334742 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.334530 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-workload-certs" pod "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" (UID: "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.334742 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.334552 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-workload-certs\") pod \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " Oct 30 20:25:34.334742 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.334577 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-workload-socket\") pod \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " Oct 30 20:25:34.334742 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.334619 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istiod-ca-cert\") pod \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " Oct 30 20:25:34.334742 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.334673 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-podinfo\") pod \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " Oct 30 20:25:34.335638 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.335278 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-kube-api-access-hwdfj\" (UniqueName: \"kubernetes.io/projected/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-kube-api-access-hwdfj\") pod \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " Oct 30 20:25:34.335638 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.335319 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fcb783a-2210-41a3-8379-b2c417ac94d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcb783a-2210-41a3-8379-b2c417ac94d8-istiod-ca-cert\") pod \"5fcb783a-2210-41a3-8379-b2c417ac94d8\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " Oct 30 20:25:34.335638 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.335361 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-token\" (UniqueName: \"kubernetes.io/projected/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-token\") pod \"5fcb783a-2210-41a3-8379-b2c417ac94d8\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " Oct 30 20:25:34.335638 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.335370 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-data" pod "5fcb783a-2210-41a3-8379-b2c417ac94d8" (UID: "5fcb783a-2210-41a3-8379-b2c417ac94d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.335638 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.335386 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-workload-socket\") pod \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " Oct 30 20:25:34.335638 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.335423 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-credential-socket\") pod \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " Oct 30 20:25:34.335638 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.335469 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istiod-ca-cert\") pod \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.335654 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-data" pod "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" (UID: "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.335662 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-token\" (UniqueName: \"kubernetes.io/projected/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-token\") pod \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.335704 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-kube-api-access-s47b8\" (UniqueName: \"kubernetes.io/projected/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-kube-api-access-s47b8\") pod \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.335737 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-workload-certs\") pod \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.335857 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-token\" (UniqueName: \"kubernetes.io/projected/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-token\") pod \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.335887 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-data\") pod \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.336015 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-podinfo\") pod \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.336056 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcb783a-2210-41a3-8379-b2c417ac94d8-kube-api-access-vwzz7\" (UniqueName: \"kubernetes.io/projected/5fcb783a-2210-41a3-8379-b2c417ac94d8-kube-api-access-vwzz7\") pod \"5fcb783a-2210-41a3-8379-b2c417ac94d8\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.336145 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-workload-socket\") pod \"5fcb783a-2210-41a3-8379-b2c417ac94d8\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.336201 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-workload-certs\") pod \"5fcb783a-2210-41a3-8379-b2c417ac94d8\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.336227 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-envoy\") pod \"5fcb783a-2210-41a3-8379-b2c417ac94d8\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.336268 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-credential-socket\") pod \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\" (UID: \"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce\") " Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.336292 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-workload-socket" pod "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" (UID: "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.336302 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-podinfo\") pod \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.336367 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-credential-socket\") pod \"5fcb783a-2210-41a3-8379-b2c417ac94d8\" (UID: \"5fcb783a-2210-41a3-8379-b2c417ac94d8\") " Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.336403 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-data\") pod \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.336437 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-token\" (UniqueName: \"kubernetes.io/projected/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-token\") pod \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\" (UID: \"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb\") " Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.336468 5047 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istiod-ca-cert\") pod \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\" (UID: \"5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4\") " Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.336553 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-workload-socket" pod "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" (UID: "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.336808 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-credential-socket" pod "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" (UID: "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.337039 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.337061 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.337055 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-workload-certs" pod "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" (UID: "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.337074 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.337108 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.337125 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.337140 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.337278 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-data" pod "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" (UID: "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.337495 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-credential-socket" pod "5fcb783a-2210-41a3-8379-b2c417ac94d8" (UID: "5fcb783a-2210-41a3-8379-b2c417ac94d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.337857 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istiod-ca-cert" pod "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" (UID: "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.338075 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-workload-certs" pod "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" (UID: "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.338080 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-credential-socket" pod "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" (UID: "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.337982 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istiod-ca-cert" pod "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" (UID: "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.338390 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fcb783a-2210-41a3-8379-b2c417ac94d8-istiod-ca-cert" pod "5fcb783a-2210-41a3-8379-b2c417ac94d8" (UID: "5fcb783a-2210-41a3-8379-b2c417ac94d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.338394 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istiod-ca-cert" pod "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" (UID: "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:34.338957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.338635 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-workload-socket" pod "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" (UID: "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.340989 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.340421 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-workload-certs" pod "5fcb783a-2210-41a3-8379-b2c417ac94d8" (UID: "5fcb783a-2210-41a3-8379-b2c417ac94d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.340989 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.340544 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-workload-socket" pod "5fcb783a-2210-41a3-8379-b2c417ac94d8" (UID: "5fcb783a-2210-41a3-8379-b2c417ac94d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.340989 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.340672 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-data" pod "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" (UID: "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.340989 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.340831 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-credential-socket" pod "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" (UID: "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.341390 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.341338 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-envoy" pod "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" (UID: "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.341390 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.341346 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-podinfo" pod "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" (UID: "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:25:34.341569 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.341505 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-envoy" pod "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" (UID: "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.341695 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.341630 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-token" pod "5fcb783a-2210-41a3-8379-b2c417ac94d8" (UID: "5fcb783a-2210-41a3-8379-b2c417ac94d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:34.342168 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.342133 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-kube-api-access-6mht8" pod "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" (UID: "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4"). InnerVolumeSpecName "kube-api-access-6mht8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:34.343501 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.343475 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcb783a-2210-41a3-8379-b2c417ac94d8-kube-api-access-vwzz7" pod "5fcb783a-2210-41a3-8379-b2c417ac94d8" (UID: "5fcb783a-2210-41a3-8379-b2c417ac94d8"). InnerVolumeSpecName "kube-api-access-vwzz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:34.343604 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.343586 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-token" pod "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" (UID: "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:34.343660 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.343642 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-kube-api-access-s47b8" pod "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" (UID: "2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb"). InnerVolumeSpecName "kube-api-access-s47b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:34.343837 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.343722 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-podinfo" pod "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" (UID: "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:25:34.343837 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.343725 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-envoy" pod "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" (UID: "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.343837 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.343802 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-podinfo" pod "5fcb783a-2210-41a3-8379-b2c417ac94d8" (UID: "5fcb783a-2210-41a3-8379-b2c417ac94d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:25:34.344544 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.344205 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-token" pod "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" (UID: "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:34.344754 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.344730 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-token" pod "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" (UID: "5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:34.344930 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.344909 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-envoy" pod "5fcb783a-2210-41a3-8379-b2c417ac94d8" (UID: "5fcb783a-2210-41a3-8379-b2c417ac94d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:34.345757 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.345736 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-podinfo" pod "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" (UID: "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:25:34.345757 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.345744 5047 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-kube-api-access-hwdfj" pod "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" (UID: "dbda9b5f-24c3-4cbc-ba7f-d134eda7adce"). InnerVolumeSpecName "kube-api-access-hwdfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:34.438571 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438457 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.438571 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438509 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.438571 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438523 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.438571 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438535 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.438571 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438548 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwzz7\" (UniqueName: \"kubernetes.io/projected/5fcb783a-2210-41a3-8379-b2c417ac94d8-kube-api-access-vwzz7\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.438571 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438563 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.438571 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438576 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438588 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438599 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438612 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438623 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb783a-2210-41a3-8379-b2c417ac94d8-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438634 5047 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-data\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438644 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438656 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438667 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438677 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438690 5047 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istio-envoy\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438701 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mht8\" (UniqueName: \"kubernetes.io/projected/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-kube-api-access-6mht8\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438713 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438725 5047 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-workload-certs\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438736 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438749 5047 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-podinfo\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438760 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwdfj\" (UniqueName: \"kubernetes.io/projected/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-kube-api-access-hwdfj\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438771 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcb783a-2210-41a3-8379-b2c417ac94d8-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438783 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcb783a-2210-41a3-8379-b2c417ac94d8-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438796 5047 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-workload-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438809 5047 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-credential-socket\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438822 5047 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce-istiod-ca-cert\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438834 5047 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4-istio-token\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.439054 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.438846 5047 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s47b8\" (UniqueName: \"kubernetes.io/projected/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb-kube-api-access-s47b8\") on node \"ip-10-0-140-235.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.561976 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.561937 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk"] Oct 30 20:25:34.564973 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:34.564942 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-xljwk"] Oct 30 20:25:35.047954 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:35.047918 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" path="/var/lib/kubelet/pods/5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4/volumes" Oct 30 20:25:35.223957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:35.223862 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" Oct 30 20:25:35.223957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:35.223876 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk" event={"ID":"5fcb783a-2210-41a3-8379-b2c417ac94d8","Type":"ContainerDied","Data":"c65f8e7be2d504dd4a801fbbed3b605c66286a7930176f8c7534c3918cf59af0"} Oct 30 20:25:35.223957 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:35.223950 5047 scope.go:122] "RemoveContainer" containerID="8f20a68f61c38434afa64fb51ff4b7758d2f38f6f132f02915d971141f61a0dd" Oct 30 20:25:35.225451 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:35.225416 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" event={"ID":"2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb","Type":"ContainerDied","Data":"be335385a0f8f9868ef6243a43cacb05e225e2e7b74c70046928bcd1088f3503"} Oct 30 20:25:35.225561 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:35.225427 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4" Oct 30 20:25:35.226849 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:35.226831 5047 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" Oct 30 20:25:35.226849 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:35.226840 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb" event={"ID":"dbda9b5f-24c3-4cbc-ba7f-d134eda7adce","Type":"ContainerDied","Data":"d35ec7f6c135167cab395cc77d74637152edcf947e48b01d4c458410ce95cf92"} Oct 30 20:25:35.234500 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:35.234481 5047 scope.go:122] "RemoveContainer" containerID="7cd34f89f7de6a092610ea8f6db53a6b8875a25a94e797a0f519993e4bb3a63a" Oct 30 20:25:35.242415 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:35.242399 5047 scope.go:122] "RemoveContainer" containerID="0f2a8b23795e8ef06cf2df563cc29498c02ad9a295d5c3c781afcd086e4bc8c8" Oct 30 20:25:35.273230 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:35.273197 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk"] Oct 30 20:25:35.285067 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:35.285031 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-frwdk"] Oct 30 20:25:35.312330 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:35.312296 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb"] Oct 30 20:25:35.318296 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:35.318266 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-mpgxb"] Oct 30 20:25:35.351763 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:35.351733 5047 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4"] Oct 30 20:25:35.353207 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:35.353185 5047 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-bb2r4"] Oct 30 20:25:37.047482 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:37.047437 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" path="/var/lib/kubelet/pods/2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb/volumes" Oct 30 20:25:37.048126 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:37.048104 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fcb783a-2210-41a3-8379-b2c417ac94d8" path="/var/lib/kubelet/pods/5fcb783a-2210-41a3-8379-b2c417ac94d8/volumes" Oct 30 20:25:37.048649 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:25:37.048610 5047 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" path="/var/lib/kubelet/pods/dbda9b5f-24c3-4cbc-ba7f-d134eda7adce/volumes" Oct 30 20:25:39.598966 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:25:39.598932 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:26:07.315887 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:07.315843 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-kdv7t_d7f89e1b-5ac7-46ce-9a5b-77a1ec2861b5/manager/0.log" Oct 30 20:26:08.036524 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:08.036492 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-nm5kw_369870ff-3389-4bb6-8b76-05a35b0cc02b/manager/0.log" Oct 30 20:26:09.600492 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:26:09.600452 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:26:10.111273 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:10.111227 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-x4r6k_540944ec-348d-46e1-836b-9aeb07d16713/cluster-monitoring-operator/0.log" Oct 30 20:26:11.249947 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:11.249915 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xbvmn_33d88d43-afaf-4899-af60-57a0020af5f1/node-exporter/0.log" Oct 30 20:26:11.374767 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:11.374740 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xbvmn_33d88d43-afaf-4899-af60-57a0020af5f1/kube-rbac-proxy/0.log" Oct 30 20:26:11.519431 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:11.519353 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xbvmn_33d88d43-afaf-4899-af60-57a0020af5f1/init-textfile/0.log" Oct 30 20:26:16.080725 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:16.080690 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-nxngq_bcf8833f-ed61-40f4-880c-2645ffae454b/networking-console-plugin/0.log" Oct 30 20:26:17.186121 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:17.186085 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/2.log" Oct 30 20:26:17.193147 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:17.193113 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6h7t5_bd7cda5e-1a4b-45f8-9cab-f96111986c08/console-operator/3.log" Oct 30 20:26:18.215108 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.215071 5047 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m"] Oct 30 20:26:18.215513 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.215491 5047 cpu_manager.go:414] "containerMap: removing container" podUID="5fcb783a-2210-41a3-8379-b2c417ac94d8" containerName="istio-proxy" Oct 30 20:26:18.215513 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.215502 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fcb783a-2210-41a3-8379-b2c417ac94d8" containerName="istio-proxy" Oct 30 20:26:18.215513 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.215513 5047 cpu_manager.go:414] "containerMap: removing container" podUID="dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" containerName="istio-proxy" Oct 30 20:26:18.215630 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.215519 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" containerName="istio-proxy" Oct 30 20:26:18.215630 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.215527 5047 cpu_manager.go:414] "containerMap: removing container" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerName="istio-proxy" Oct 30 20:26:18.215630 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.215532 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerName="istio-proxy" Oct 30 20:26:18.215630 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.215541 5047 cpu_manager.go:414] "containerMap: removing container" podUID="b0d053db-3c2e-4701-b045-68a75abeaa89" containerName="maas-api" Oct 30 20:26:18.215630 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.215546 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d053db-3c2e-4701-b045-68a75abeaa89" containerName="maas-api" Oct 30 20:26:18.215630 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.215577 5047 cpu_manager.go:414] "containerMap: removing container" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerName="istio-proxy" Oct 30 20:26:18.215630 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.215582 5047 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerName="istio-proxy" Oct 30 20:26:18.215853 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.215645 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="2834fba3-6ed1-4b0c-ad61-bf9e41da5bbb" containerName="istio-proxy" Oct 30 20:26:18.215853 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.215656 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fcb783a-2210-41a3-8379-b2c417ac94d8" containerName="istio-proxy" Oct 30 20:26:18.215853 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.215662 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbda9b5f-24c3-4cbc-ba7f-d134eda7adce" containerName="istio-proxy" Oct 30 20:26:18.215853 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.215669 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="5671acbc-24a4-46dd-aa3f-aa7a25b6c8d4" containerName="istio-proxy" Oct 30 20:26:18.215853 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.215674 5047 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d053db-3c2e-4701-b045-68a75abeaa89" containerName="maas-api" Oct 30 20:26:18.220237 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.220209 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:18.237195 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.237160 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m"] Oct 30 20:26:18.246182 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.246148 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wjhm6\"/\"kube-root-ca.crt\"" Oct 30 20:26:18.246182 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.246173 5047 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wjhm6\"/\"openshift-service-ca.crt\"" Oct 30 20:26:18.246387 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.246245 5047 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wjhm6\"/\"default-dockercfg-st55s\"" Oct 30 20:26:18.345110 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.345065 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxszv\" (UniqueName: \"kubernetes.io/projected/498e045b-3d87-4c4b-9a0f-98df10929a16-kube-api-access-kxszv\") pod \"perf-node-gather-daemonset-8kl8m\" (UID: \"498e045b-3d87-4c4b-9a0f-98df10929a16\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:18.345312 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.345133 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/498e045b-3d87-4c4b-9a0f-98df10929a16-lib-modules\") pod \"perf-node-gather-daemonset-8kl8m\" (UID: \"498e045b-3d87-4c4b-9a0f-98df10929a16\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:18.345312 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.345237 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/498e045b-3d87-4c4b-9a0f-98df10929a16-proc\") pod \"perf-node-gather-daemonset-8kl8m\" (UID: \"498e045b-3d87-4c4b-9a0f-98df10929a16\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:18.345312 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.345274 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/498e045b-3d87-4c4b-9a0f-98df10929a16-podres\") pod \"perf-node-gather-daemonset-8kl8m\" (UID: \"498e045b-3d87-4c4b-9a0f-98df10929a16\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:18.345312 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.345295 5047 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/498e045b-3d87-4c4b-9a0f-98df10929a16-sys\") pod \"perf-node-gather-daemonset-8kl8m\" (UID: \"498e045b-3d87-4c4b-9a0f-98df10929a16\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:18.446755 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.446723 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxszv\" (UniqueName: \"kubernetes.io/projected/498e045b-3d87-4c4b-9a0f-98df10929a16-kube-api-access-kxszv\") pod \"perf-node-gather-daemonset-8kl8m\" (UID: \"498e045b-3d87-4c4b-9a0f-98df10929a16\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:18.446964 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.446770 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/498e045b-3d87-4c4b-9a0f-98df10929a16-lib-modules\") pod \"perf-node-gather-daemonset-8kl8m\" (UID: \"498e045b-3d87-4c4b-9a0f-98df10929a16\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:18.446964 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.446803 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/498e045b-3d87-4c4b-9a0f-98df10929a16-proc\") pod \"perf-node-gather-daemonset-8kl8m\" (UID: \"498e045b-3d87-4c4b-9a0f-98df10929a16\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:18.446964 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.446850 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/498e045b-3d87-4c4b-9a0f-98df10929a16-podres\") pod \"perf-node-gather-daemonset-8kl8m\" (UID: \"498e045b-3d87-4c4b-9a0f-98df10929a16\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:18.446964 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.446868 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/498e045b-3d87-4c4b-9a0f-98df10929a16-proc\") pod \"perf-node-gather-daemonset-8kl8m\" (UID: \"498e045b-3d87-4c4b-9a0f-98df10929a16\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:18.446964 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.446891 5047 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/498e045b-3d87-4c4b-9a0f-98df10929a16-sys\") pod \"perf-node-gather-daemonset-8kl8m\" (UID: \"498e045b-3d87-4c4b-9a0f-98df10929a16\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:18.446964 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.446928 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/498e045b-3d87-4c4b-9a0f-98df10929a16-podres\") pod \"perf-node-gather-daemonset-8kl8m\" (UID: \"498e045b-3d87-4c4b-9a0f-98df10929a16\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:18.447263 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.446989 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/498e045b-3d87-4c4b-9a0f-98df10929a16-lib-modules\") pod \"perf-node-gather-daemonset-8kl8m\" (UID: \"498e045b-3d87-4c4b-9a0f-98df10929a16\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:18.447263 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.446999 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/498e045b-3d87-4c4b-9a0f-98df10929a16-sys\") pod \"perf-node-gather-daemonset-8kl8m\" (UID: \"498e045b-3d87-4c4b-9a0f-98df10929a16\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:18.465207 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.465139 5047 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxszv\" (UniqueName: \"kubernetes.io/projected/498e045b-3d87-4c4b-9a0f-98df10929a16-kube-api-access-kxszv\") pod \"perf-node-gather-daemonset-8kl8m\" (UID: \"498e045b-3d87-4c4b-9a0f-98df10929a16\") " pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:18.533378 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.533333 5047 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:18.824298 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:18.824262 5047 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m"] Oct 30 20:26:18.825299 ip-10-0-140-235 kubenswrapper[5047]: W1030 20:26:18.825274 5047 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod498e045b_3d87_4c4b_9a0f_98df10929a16.slice/crio-96048902dbbc6daf29c67c2bee698a5f08d294a45c46c793d1a901be70cab273 WatchSource:0}: Error finding container 96048902dbbc6daf29c67c2bee698a5f08d294a45c46c793d1a901be70cab273: Status 404 returned error can't find the container with id 96048902dbbc6daf29c67c2bee698a5f08d294a45c46c793d1a901be70cab273 Oct 30 20:26:19.383016 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:19.382969 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-jdc67_969393ae-3d0e-4f83-bf03-0396c386ffbf/volume-data-source-validator/0.log" Oct 30 20:26:19.453447 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:19.453406 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" event={"ID":"498e045b-3d87-4c4b-9a0f-98df10929a16","Type":"ContainerStarted","Data":"67ab12873891d64f5a0e149c6d37ec88fa08baf491f2e78ff86b36c8ea8de506"} Oct 30 20:26:19.453625 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:19.453467 5047 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" event={"ID":"498e045b-3d87-4c4b-9a0f-98df10929a16","Type":"ContainerStarted","Data":"96048902dbbc6daf29c67c2bee698a5f08d294a45c46c793d1a901be70cab273"} Oct 30 20:26:19.453625 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:19.453497 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:19.489625 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:19.489567 5047 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" podStartSLOduration=1.4895555200000001 podStartE2EDuration="1.48955552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:19.488872656 +0000 UTC m=+4932.563328015" watchObservedRunningTime="2026-10-30 20:26:19.48955552 +0000 UTC m=+4932.564010875" Oct 30 20:26:21.427696 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:21.427658 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jqrww_a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b/dns/0.log" Oct 30 20:26:21.548905 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:21.548867 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jqrww_a4b2582d-0fc5-4ab8-94aa-6ac1310b3f6b/kube-rbac-proxy/0.log" Oct 30 20:26:22.411915 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:22.411882 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xpxgd_415a1d40-3275-4624-98cd-3df86bbc819a/dns-node-resolver/0.log" Oct 30 20:26:23.768155 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:23.768122 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tbttr_830ddb4e-a8f8-406c-9e89-755bb0d48e0e/node-ca/0.log" Oct 30 20:26:25.468349 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:25.468321 5047 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wjhm6/perf-node-gather-daemonset-8kl8m" Oct 30 20:26:31.028048 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:31.027965 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kzmjv_7d62bc31-309d-4724-8bda-83f24f229a2e/discovery/0.log" Oct 30 20:26:31.702327 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:31.702297 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c7d47d6c9-jvqgk_eee3ec1c-3cb4-46c8-9e8b-25ca31925e6d/kube-auth-proxy/0.log" Oct 30 20:26:31.949650 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:31.949615 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-mptvz_645e5088-e265-4836-a447-1cb2b21cc050/payload-pre-processing/0.log" Oct 30 20:26:32.076742 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:32.076652 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-rspbb_8fa7fdd5-24e6-4482-9bcc-91fe3c45b92a/payload-processing/0.log" Oct 30 20:26:32.215572 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:32.215538 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77d478f789-d787c_20403b1e-e479-45b9-bb85-52cf10725d86/router/0.log" Oct 30 20:26:34.872401 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:34.872365 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ccd7w_3e626b03-28b0-4840-bbe2-518834311815/serve-healthcheck-canary/0.log" Oct 30 20:26:35.925981 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:35.925944 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-wd8lm_11d31a53-ae1b-4d46-aa8c-6c00e89ed539/insights-operator/1.log" Oct 30 20:26:35.929495 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:35.929458 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-wd8lm_11d31a53-ae1b-4d46-aa8c-6c00e89ed539/insights-operator/2.log" Oct 30 20:26:36.051023 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:36.050988 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5rvfx_5557e025-30b7-4315-b7a1-308e166c9a3b/kube-rbac-proxy/0.log" Oct 30 20:26:36.174033 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:36.173983 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5rvfx_5557e025-30b7-4315-b7a1-308e166c9a3b/exporter/0.log" Oct 30 20:26:36.299450 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:36.299363 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5rvfx_5557e025-30b7-4315-b7a1-308e166c9a3b/extractor/0.log" Oct 30 20:26:37.132694 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:37.132656 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-4sdcp-kxxdm_770e23c1-db57-4590-a46d-77ff7f9909ea/insights-gathering/0.log" Oct 30 20:26:39.601973 ip-10-0-140-235 kubenswrapper[5047]: E1030 20:26:39.601938 5047 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5rvfx" containerName="extractor" Oct 30 20:26:40.245247 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:40.245212 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-84dd667bd6-hz7tl_3977e82d-a7b6-4f25-aa74-8d1f6488ff41/manager/0.log" Oct 30 20:26:41.636838 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:41.636804 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65cb454cc4-5lnl7_70ddcddf-b107-4a4a-ac56-4b451ead2612/manager/0.log" Oct 30 20:26:42.481776 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:42.481740 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-564b7_0f369006-b3bf-412c-9c5b-90e2f945bc76/manager/0.log" Oct 30 20:26:42.853669 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:42.853635 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-ch865_dfe5c738-462f-461b-b1e9-1f449896ed4e/manager/0.log" Oct 30 20:26:42.979268 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:42.979240 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8589ff54bc-ch865_dfe5c738-462f-461b-b1e9-1f449896ed4e/copy-manifests/0.log" Oct 30 20:26:43.486993 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:43.486950 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-vz8vl_a703457c-cd6f-4b39-b8db-e7c3c419e72d/proxy/0.log" Oct 30 20:26:45.515332 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:45.515298 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-8465994787-q2thr_21e9ae46-ccd9-410e-ad31-f13872b3ab36/openshift-lws-operator/0.log" Oct 30 20:26:53.787934 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:53.787895 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-6zk7f_452b3edc-e035-446c-9b7b-ee8154f3c6c3/migrator/0.log" Oct 30 20:26:53.914020 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:53.913974 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-lp2xk_facfc926-9288-41c4-bdc3-5d8acf446292/migrator/0.log" Oct 30 20:26:54.783733 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:54.783698 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l_721cf657-76b1-4fdf-868d-d84cb9fa0b14/kube-storage-version-migrator-operator/1.log" Oct 30 20:26:54.785196 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:54.785169 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-nkm5l_721cf657-76b1-4fdf-868d-d84cb9fa0b14/kube-storage-version-migrator-operator/0.log" Oct 30 20:26:58.564219 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:58.564190 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k28t4_7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad/kube-multus-additional-cni-plugins/0.log" Oct 30 20:26:58.690250 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:58.690219 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k28t4_7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad/egress-router-binary-copy/0.log" Oct 30 20:26:58.816684 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:58.816612 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k28t4_7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad/cni-plugins/0.log" Oct 30 20:26:58.942630 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:58.942602 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k28t4_7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad/bond-cni-plugin/0.log" Oct 30 20:26:59.069692 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:59.069622 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k28t4_7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad/routeoverride-cni/0.log" Oct 30 20:26:59.196078 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:59.196052 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k28t4_7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad/whereabouts-cni-bincopy/0.log" Oct 30 20:26:59.322025 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:59.321939 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k28t4_7fe5ab76-2e1c-4dda-bfb8-6bf9c48dccad/whereabouts-cni/0.log" Oct 30 20:26:59.544759 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:26:59.544730 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-kgvdt_ceff794b-7638-49c7-a0f4-495a8cf3d8c7/kube-multus/0.log" Oct 30 20:27:00.137749 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:27:00.137720 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-frj4v_93f0a70e-357e-4d65-b554-42363280bebd/network-metrics-daemon/0.log" Oct 30 20:27:00.248924 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:27:00.248894 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-frj4v_93f0a70e-357e-4d65-b554-42363280bebd/kube-rbac-proxy/0.log" Oct 30 20:27:02.564168 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:27:02.564123 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qftp_f7537906-9d0e-4b31-9d76-45936780d55c/ovn-controller/0.log" Oct 30 20:27:02.700132 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:27:02.700098 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qftp_f7537906-9d0e-4b31-9d76-45936780d55c/ovn-acl-logging/0.log" Oct 30 20:27:02.829985 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:27:02.829957 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qftp_f7537906-9d0e-4b31-9d76-45936780d55c/kube-rbac-proxy-node/0.log" Oct 30 20:27:02.959982 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:27:02.959951 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qftp_f7537906-9d0e-4b31-9d76-45936780d55c/kube-rbac-proxy-ovn-metrics/0.log" Oct 30 20:27:03.087341 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:27:03.087262 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qftp_f7537906-9d0e-4b31-9d76-45936780d55c/northd/0.log" Oct 30 20:27:03.215114 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:27:03.215084 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qftp_f7537906-9d0e-4b31-9d76-45936780d55c/nbdb/0.log" Oct 30 20:27:03.341935 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:27:03.341861 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qftp_f7537906-9d0e-4b31-9d76-45936780d55c/sbdb/0.log" Oct 30 20:27:03.712379 ip-10-0-140-235 kubenswrapper[5047]: I1030 20:27:03.712275 5047 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qftp_f7537906-9d0e-4b31-9d76-45936780d55c/ovnkube-controller/0.log"